- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DocAIGen, a framework for generating AI-modified documents to evaluate detection and localization methods. The authors further propose DocAIdet, a method designed to detect and localize AI-modified content in documents.
Critical strengths
- The proposed dataset is potentially valuable for future research on AI-modified document detection and localization.
Critical weaknesses
-
The quality and representativeness of the proposed dataset raise several concerns. First, the dataset is constructed using relatively small and outdated open-source generative models, such as Llama-3.2-3B and Stable Diffusion v1.5. This raises questions about whether the generated documents adequately reflect the capabilities of more advanced generative models, making it questionable whether the reported findings still holds on more advanced models. Second, the paper lacks sufficient qualitative examples to demonstrate the quality of the generated documents. The examples provided in the Appendix are too small to be clearly examined. Furthermore, no human evaluation is conducted to assess the realism and quality of the generated documents. Consequently, claims such as "no shortcuts" are not sufficiently supported. Although the original and modified documents are rendered using the same engine, this does not necessarily eliminate potential formatting shortcuts. For example, AI-modified text may be excessively long, resulting in unnatural page layouts that could be exploited by detectors.
-
The experimental evaluation is not sufficiently comprehensive, and the generalization ability of the proposed method remains unclear. In particular, a piecewise baseline that separately applies text and image detectors should be one of the most important comparisons. However, the paper does not clearly specify which text and image detectors are used in this baseline, making it difficult to assess the fairness and strength of the comparison. Moreover, the evaluated text and image detectors appear to be relatively simple fine-tuned models, while stronger and more representative methods, such as DRCT, AIDE, and DDA for image detection, are not included. This limits the persuasiveness of the experimental conclusions. Additionally, Gemini-3-Flash (a model released near a year ago) already achieves strong performance, suggesting that the advantages of the proposed method over capable general-purpose models require further investigation. Another major concern is the generalization ability of DocAIdet. The proposed method is trained on a subset of the constructed benchmark, which raises the possibility of learning dataset-specific shortcuts. Even if the generators used for training and testing differ, the underlying data construction pipeline remains the same, potentially introducing shared artifacts or formatting patterns that the detector could exploit. The design of DocAIdet further motivates this concern. Given the relatively limited capabilities of its SmolVLM backbone, its strong performance in detecting AI-modified text rendered within document images is somewhat surprising and warrants further investigation. While this observation does not necessarily indicate overfitting, it raises questions about whether the model has learned generalizable detection features or relies on dataset-specific artifacts.
Questions for authors
-
Could the authors provide additional qualitative and quantitative evaluations to demonstrate the quality, realism, and reliability of the proposed benchmark? In particular, a human evaluation assessing document realism and potential formatting shortcuts would be valuable.
-
Could the authors incorporate stronger and more representative AI-generated text and image detectors, particularly in the piecewise baseline, to provide a more comprehensive and fair comparison?
-
Could the authors provide stronger evidence of the generalization ability of DocAIdet? For example, evaluating the method on independently collected real-world documents with disclosed AI usage, rather than documents generated through the same construction pipeline, would help assess whether the method generalizes beyond dataset-specific artifacts.