- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper first introduces AIGDoc-Pilot to investigate the potential limitations of existing AI-generated image detectors when applied to AI-generated document images. The authors further propose AIGDoc, a large-scale benchmark designed to comprehensively evaluate the performance of existing detectors in this setting.
Critical strengths
- The identified problems faced by existing detectors on AI-generated document images are potentially valuable and could provide useful insights for future research.
Critical weaknesses
- The experimental analysis is not sufficiently comprehensive to support the reported findings. Most of the analyses are conducted on AIGDoc-Pilot, which is relatively small and contains only images generated through image-to-image generation. Furthermore, the number of evaluated detectors is quite limited, with several important families of detection methods being overlooked. These include simple classifiers based on pretrained models such as CLIP, which has been shown to exhibit strong robustness in AIGIBench, as well as methods that leverage frequency-domain information, such as FreqNet, AIDE, and SAFE. In addition, all generated images are saved in JPEG format, which may introduce format-specific biases. Since PNG is also a widely used image format, it remains unclear whether the reported findings generalize to images stored in other formats or are partially influenced by the specific compression settings.
Questions for authors
I have concerns about the overall organization of the benchmark and the experimental analysis. For a benchmark-oriented study, I would generally expect the authors to first establish a comprehensive benchmark and then derive findings and conclusions from experiments conducted on that benchmark. However, in the current paper, most of the findings are based on experiments conducted on a relatively small dataset with a limited selection of detectors. In contrast, the proposed large-scale AIGDoc benchmark appears to be primarily used for evaluating overall detection performance, with relatively limited additional analysis. This organization makes it difficult to understand why the main findings are derived from AIGDoc-Pilot rather than the more comprehensive AIGDoc benchmark. More importantly, it raises concerns about the reliability and generalizability of the conclusions presented in Section 3.
I would therefore suggest conducting the analyses in Section 3 on the full AIGDoc benchmark, incorporating a broader range of representative detectors. This would not only provide stronger empirical support for the reported findings but also improve the clarity and logical flow of the paper. In particular, introducing the benchmark before presenting the main experimental findings would make the paper easier to follow and help readers better understand how the proposed benchmark supports the authors' conclusions.
4 comments
1. Expanded baseline evaluation.
We have expanded the evaluation on the full AIGDoc benchmark by adding representative detectors from complementary families, including the CLIP-based UnivFD and frequency-oriented methods FreqNet, SAFE, AIDE, and SPAI. The corresponding Acc. and AUC results are included in the revised Tables 5 and 9. As shown in Table I & II below, despite these stronger and more diverse baselines, PE-LoRA$^\ddagger$ remains the best method in terms of mean Acc., supporting the effectiveness of document-aware training and region selection on the full benchmark.
Table I. Detection Accuracy across AIGDoc Settings.
| Generator | UnivFD | DRCT | FreqNet | SAFE | AIDE | Effort | SPAI | B-Free | DDA | MIRROR | OmniAID | PROBE | SICA | PE-LoRA | PE-LoRA$^\dagger$ | PE-LoRA$^\ddagger$ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text2Image | ||||||||||||||||
| Z-Image-Turbo | 49.4 | 61.8 | 49.3 | 50.0 | 49.3 | 85.2 | 74.9 | 52.3 | 61.4 | 49.9 | 49.8 | 55.1 | 44.9 | 68.0 | 92.8 | 90.5 |
| HiDream-O1-Image | 56.5 | 62.1 | 84.6 | 50.0 | 50.3 | 72.7 | 48.5 | 97.0 | 95.2 | 66.8 | 63.2 | 97.9 | 75.8 | 95.2 | 97.9 | 93.5 |
| Seedream4.5 | 50.1 | 51.7 | 64.0 | 50.5 | 48.3 | 58.2 | 75.9 | 95.2 | 91.8 | 52.8 | 53.3 | 99.5 | 61.8 | 98.0 | 97.4 | 95.7 |
| Nano-Banana-2 | 50.0 | 46.2 | 56.0 | 50.0 | 48.9 | 60.2 | 56.1 | 66.0 | 58.9 | 50.4 | 51.0 | 63.9 | 55.4 | 59.8 | 67.5 | 60.5 |
| GPT-Image-2 | 50.0 | 49.5 | 52.7 | 49.9 | 48.5 | 74.2 | 59.0 | 51.3 | 47.0 | 49.8 | 50.3 | 51.2 | 47.4 | 49.4 | 59.7 | 83.9 |
| Image-Edit | ||||||||||||||||
| FLUX.1-Kontext | 49.8 | 51.2 | 51.1 | 50.0 | 45.0 | 54.5 | 62.5 | 49.6 | 53.7 | 50.0 | 51.2 | 50.7 | 51.9 | 51.5 | 58.9 | 93.5 |
| FLUX.2-Klein | 49.5 | 43.7 | 52.6 | 50.3 | 44.3 | 48.3 | 46.2 | 50.7 | 52.1 | 50.0 | 50.5 | 51.1 | 52.0 | 50.2 | 53.3 | 83.9 |
| Hunyuan-3-Instruct | 49.8 | 57.0 | 65.3 | 50.0 | 44.1 | 61.1 | 74.7 | 72.4 | 95.8 | 54.2 | 66.0 | 96.6 | 64.5 | 66.4 | 97.0 | 99.5 |
| LongCat-Image-Edit | 49.8 | 47.9 | 57.7 | 50.0 | 47.7 | 53.3 | 61.7 | 71.9 | 68.0 | 50.4 | 53.1 | 69.1 | 56.8 | 75.7 | 84.2 | 99.2 |
| OmniGen2 | 56.1 | 57.2 | 52.8 | 50.3 | 61.7 | 58.1 | 56.7 | 81.3 | 65.7 | 72.4 | 66.9 | 73.8 | 68.5 | 72.2 | 83.4 | 94.1 |
| Qwen-Image-Edit | 51.0 | 49.8 | 56.1 | 50.0 | 48.4 | 57.2 | 52.5 | 82.2 | 78.9 | 53.7 | 56.4 | 81.8 | 60.4 | 73.3 | 89.8 | 97.3 |
| Step1X-Edit | 49.6 | 55.5 | 58.4 | 49.9 | 44.9 | 59.0 | 75.3 | 74.3 | 76.2 | 50.6 | 53.7 | 78.5 | 55.5 | 75.1 | 82.3 | 84.5 |
| Seedream4.5 | 51.4 | 62.1 | 54.8 | 49.9 | 44.2 | 54.8 | 82.7 | 97.2 | 94.7 | 50.2 | 53.0 | 99.2 | 53.5 | 97.1 | 99.2 | 99.5 |
| Seedream5.0-Lite | 52.8 | 62.1 | 55.6 | 49.9 | 44.0 | 48.3 | 84.0 | 95.6 | 85.8 | 50.3 | 52.7 | 98.9 | 54.4 | 93.6 | 98.0 | 98.1 |
| Nano-Banana-2 | 49.3 | 45.0 | 49.6 | 49.9 | 44.3 | 54.9 | 66.8 | 49.0 | 50.8 | 50.0 | 54.4 | 50.4 | 51.3 | 50.2 | 57.4 | 60.6 |
| Digital-Image-Edit | ||||||||||||||||
| Qwen-Image-Edit | 47.8 | 34.9 | 50.0 | 50.0 | 22.0 | 50.1 | 50.0 | 30.6 | 23.9 | 50.0 | 50.0 | 39.1 | 8.3 | 55.9 | 75.4 | 65.6 |
| Nano-Banana-2 | 48.1 | 36.8 | 50.0 | 50.1 | 19.1 | 24.5 | 64.5 | 17.5 | 24.6 | 51.1 | 50.0 | 36.1 | 50.9 | 49.5 | 50.1 | 50.0 |
| Average | 50.6 | 51.4 | 56.5 | 50.0 | 44.4 | 57.3 | 64.2 | 66.7 | 66.1 | 53.1 | 54.4 | 70.2 | 53.7 | 69.5 | 79.1 | 85.3 |
Table II. Detection AUC across AIGDoc Settings.
| Generator | UnivFD | DRCT | FreqNet | SAFE | AIDE | Effort | SPAI | B-Free | DDA | MIRROR | OmniAID | PROBE | SICA | PE-LoRA | PE-LoRA$^\dagger$ | PE-LoRA$^\ddagger$ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text2Image | ||||||||||||||||
| Z-Image-Turbo | 48.0 | 69.5 | 79.4 | 79.6 | 78.3 | 93.2 | 81.1 | 60.9 | 78.4 | 56.4 | 31.8 | 86.1 | 42.9 | 84.2 | 97.7 | 97.2 |
| HiDream-O1-Image | 82.6 | 66.7 | 96.7 | 88.2 | 82.8 | 83.5 | 46.7 | 99.6 | 99.1 | 93.5 | 87.1 | 99.9 | 78.9 | 99.3 | 99.9 | 98.7 |
| Seedream4.5 | 67.8 | 51.4 | 89.2 | 70.2 | 66.8 | 64.0 | 91.4 | 99.0 | 97.8 | 67.2 | 41.0 | 100.0 | 64.8 | 99.6 | 99.9 | 99.8 |
| Nano-Banana-2 | 55.9 | 45.2 | 69.6 | 53.4 | 66.6 | 63.6 | 58.5 | 70.6 | 63.6 | 67.8 | 47.0 | 78.5 | 57.5 | 58.2 | 79.0 | 64.5 |
| GPT-Image-2 | 67.7 | 51.0 | 76.9 | 52.1 | 56.2 | 83.3 | 61.0 | 47.3 | 46.9 | 64.4 | 38.2 | 75.8 | 48.3 | 35.8 | 72.7 | 94.3 |
| Image-Edit | ||||||||||||||||
| FLUX.1-Kontext | 46.8 | 53.2 | 46.9 | 63.5 | 50.5 | 58.5 | 66.8 | 47.6 | 72.4 | 54.3 | 54.5 | 71.4 | 58.9 | 61.3 | 68.6 | 99.4 |
| FLUX.2-Klein | 43.5 | 42.4 | 56.6 | 65.6 | 48.4 | 36.9 | 40.5 | 44.9 | 54.9 | 50.3 | 54.0 | 51.3 | 58.4 | 41.7 | 41.2 | 91.8 |
| Hunyuan-3-Instruct | 53.3 | 62.1 | 89.4 | 80.0 | 51.5 | 72.6 | 81.4 | 87.3 | 99.4 | 74.8 | 66.8 | 99.7 | 78.7 | 86.6 | 99.7 | 100.0 |
| LongCat-Image-Edit | 44.1 | 49.0 | 65.0 | 65.9 | 54.0 | 55.9 | 65.5 | 87.4 | 87.1 | 61.2 | 58.8 | 89.5 | 70.5 | 94.6 | 96.6 | 100.0 |
| OmniGen2 | 68.8 | 62.9 | 66.2 | 68.4 | 73.9 | 67.2 | 57.6 | 85.4 | 75.2 | 82.7 | 70.6 | 90.3 | 76.9 | 92.1 | 90.3 | 99.3 |
| Qwen-Image-Edit | 55.9 | 51.2 | 74.5 | 63.3 | 62.6 | 64.8 | 49.2 | 92.1 | 94.0 | 68.8 | 72.0 | 96.4 | 70.6 | 89.7 | 97.0 | 99.8 |
| Step1X-Edit | 42.1 | 58.0 | 66.8 | 75.1 | 48.6 | 66.7 | 82.9 | 80.0 | 84.9 | 58.4 | 61.3 | 88.0 | 67.7 | 85.9 | 83.0 | 97.0 |
| Seedream4.5 | 56.3 | 69.1 | 78.3 | 52.6 | 50.0 | 68.5 | 92.0 | 99.4 | 99.4 | 59.5 | 52.4 | 100.0 | 64.0 | 99.8 | 100.0 | 100.0 |
| Seedream5.0-Lite | 62.1 | 68.1 | 76.0 | 46.3 | 49.2 | 58.1 | 96.2 | 98.7 | 97.4 | 57.7 | 52.2 | 99.9 | 63.7 | 99.2 | 99.8 | 99.8 |
| Nano-Banana-2 | 32.9 | 44.2 | 36.0 | 35.7 | 48.4 | 67.0 | 71.4 | 48.1 | 63.4 | 60.7 | 57.7 | 74.8 | 56.4 | 56.2 | 86.6 | 85.5 |
| Digital-Image-Edit | ||||||||||||||||
| Qwen-Image-Edit | 36.9 | 29.5 | 99.9 | 34.4 | 8.7 | 48.1 | 98.6 | 19.4 | 14.2 | 6.3 | 10.2 | 2.0 | 2.5 | 77.0 | 91.2 | 99.2 |
| Nano-Banana-2 | 3.8 | 30.0 | 27.3 | 26.1 | 0.6 | 14.9 | 99.8 | 15.6 | 13.3 | 61.3 | 54.0 | 0.0 | 57.0 | 11.2 | 28.9 | 13.3 |
| Average | 51.1 | 53.1 | 70.3 | 60.0 | 52.8 | 62.8 | 73.0 | 69.6 | 73.0 | 61.5 | 53.5 | 76.7 | 59.9 | 74.8 | 84.2 | 90.6 |
2. AIGDoc-Pilot is an analytical dataset under controlled conditions.
AIGDoc-Pilot serves a different purpose from AIGDoc. It is a controlled diagnostic testbed rather than a substitute for a comprehensive benchmark. To study whether document images introduce a distinct detection domain gap, we must control for confounding factors such as the generation model, generation procedure, and semantic content. AIGDoc-Pilot therefore uses aligned generation settings, in which generated samples are produced through image-to-image generation or VAE reconstruction. This design enables paired comparisons that isolate the effect of the document domain. In contrast, AIGDoc intentionally covers diverse generators and three realistic generation scenarios; this diversity is essential for evaluating generalization, but makes causal attribution to a single factor much more difficult.
We also clarify why Sec. 3 focuses on a limited set of detectors. The purpose of AIGDoc-Pilot is diagnostic analysis. We therefore selected representative detectors with relatively strong robustness under JPEG95 compression, including B-Free, DDA, MIRROR, PROBE, and OmniAID. These methods provide sufficiently non-degenerate responses for examining systematic changes across domains, spatial locations, and region densities.
In contrast, many detectors exhibit near-chance accuracy after JPEG compression, as also reported in AIGIBench [1]. When a detector produces nearly constant or highly unstable predictions, its apparent variation across regions or domains is dominated by threshold instability and does not provide reliable evidence for identifying systematic trends. Nevertheless, we agree that this controlled analysis alone is insufficient for evaluating overall performance. We have therefore expanded the full AIGDoc benchmark comparison to include a broader range of detectors, including UnivFD, FreqNet, SAFE, AIDE, and SPAI, as reported in Tables 5 and 9 of the manuscript.
3. The experimental results for AIGDoc validate the discovery mechanism.
The full AIGDoc results provide large-scale validation of the controlled findings. Across its 17 generation settings, text-density-guided region selection improves PE-LoRA from 69.5% to 79.1% mean Acc. (+9.6 percentage points). Training with document-domain data further improves the mean Acc. to 85.3% (+6.2 percentage points). These results show that the observations from AIGDoc-Pilot are not used in isolation: they motivate the design choices, which are then validated under the substantially broader and more realistic AIGDoc benchmark.
4. The essence of JPEG compression is to avoid the detection shortcuts introduced by PNG.
JPEG compression is used to reduce format-induced shortcut bias. In the document-image workflows considered in this work, authentic images are commonly camera-captured or distributed after compression, whereas generated images are frequently available as lossless PNG files. Directly comparing compressed real images with uncompressed PNG-generated images can allow a detector to exploit file-format and compression differences rather than generation artifacts. We therefore evaluate generated PNG images after JPEG95 compression, following the practical evaluation protocol also adopted by MIRROR [2].
Reference
[1] Li Z, Yan J, He Z, et al. Is artificial intelligence generated image detection a solved problem?[J]. Advances in neural information processing systems, 2025, 38.
[2] Liu R, Cui M, Qin Z, et al. Mirror: Manifold ideal reference reconstructor for generalizable ai-generated image detection.[J]. Advances in neural information processing systems, 2026.
I appreciate the authors' detailed responses. However, my major concerns remain unaddressed.
My primary concern is that this work is fundamentally a benchmark- and analysis-oriented study. However, the main analytical conclusions are drawn from a relatively small subset with limited evaluation, while the larger benchmark is used primarily for overall performance evaluation. This makes the analytical findings, and consequently the claimed analytical contribution of the work, less convincing.
The authors mention in the rebuttal that their findings on text density motivated the design of the improved cropping strategy, which they demonstrate to be effective. However, two important questions remain unanswered.
First, the authors report three key findings, yet the proposed method only builds upon one of them. How are the other two findings further validated, investigated, or translated into practical implications? Without further exploration, it remains unclear how much of the analytical contribution is actually substantiated by the subsequent method design.
Second, the proposed method lacks sufficient ablation and comparison with alternative cropping strategies. The current experiments only compare it against strict center cropping. What about random cropping, or resizing the image instead of cropping it? These are important comparisons for establishing whether the proposed text-density-based strategy itself is effective, rather than merely outperforming a potentially suboptimal baseline.
More concerningly, according to the results in Table I of the response to Reviewer XvYE, incorporating text-density-based cropping also improves performance on general AI-image detection, where text is less prevalent. This raises another possibility: the observed improvement may stem primarily from the limitations of strict center cropping, rather than from the specific effectiveness of text-density-based cropping. Without comparisons against alternative cropping and resizing strategies, it is difficult to distinguish between these explanations.
To summarize, the additional experimental results provided in the rebuttal do not directly address the questions I raised. Specifically, these experiments continue to use the entire AIGDoc benchmark for overall performance evaluation across more detectors, whereas my primary concern is the validity and robustness of the fine-grained analysis conducted on AIGDoc-Pilot in Section 3. These are fundamentally different aspects of the work, and the additional overall evaluation does not resolve my concerns regarding the latter. Therefore, I maintain my original score.
We thank the reviewer for these closely related and constructive comments. We agree that the original presentation did not sufficiently distinguish the roles of AIGDoc-Pilot and AIGDoc, nor did it make the validation chain from controlled analysis to large-scale benchmarking sufficiently explicit.