- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the effect of the residual (reconstruction error of an autoencoder) on fake image detectors. They observe that both the magnitude as well as the sign of this residual causes detectors to misclassify images. This is practically significant since, most detectors have are trained on generators with a certain residual magnitude which often decreases with future models, making reliance on these cues undesirable. The paper also claims that within vision transformer based detection backbones, the patch size is a major contributor in the detection of these results, more importantly. a larger patch size results in these details being lost. Based on these findings, the authors propose 2 changes, (i) halving the patch size, (ii) residual augmentation which involves using images that have been reconstructed once or twice with the latent diffusion autoencoder and with some probability adding the residual at a different magnitude. The authors report results on several benchmarks, such as X-AIGD, newT2I, GenImage, Synthbuster, AlignedForensics etc reporting a wide array of results demonstrating the promise of their approach.
Critical strengths
- The data augmentation style is novel to the best of my knowledge. It also seems to result in good generalization.
- The breadth of evaluation is impressive, several recent models, across multiple benchmarks. I also really like the fact that the authors have run the experiment where each baseline is compared to at its original setting as well as when training with their proposed SA-1B. Some of the experiments are also reported over 3 seeds which is a good degree of rigor.
- The ablation coverage is also good, demonstrating the effectiveness the design choices (based on balanced accuracy)
Critical weaknesses
- I personally feel the evidence behind the smaller patch sizes helping could be more convincing. My main concerns arise from Fig 4, where there is not a huge difference in terms of accuracy between the smaller patch size (around 6,7 and the baseline one, around 12). The difference could be experimental noise. The evidence in Table 2, makes this more convincing to me, plus it also intuitively makes sense to me, however, the original experiment in Fig 4 should be more convincing. I also have concerns with the use of balanced accuracy which also relates to these findings.
- The paper makes some crucial arguments using balanced accuracy (Fig 2,4,6 and also Table 2). I feel threshold-free estimate should be made in order to make these conclusions. For instance, in Table 1, baseline methods Stay-Positive, AlignedForensic, PROBE etc show non-trivial improvement (AUROC is good but accuracy is bad) suggesting that some of the differences may be inflated by the choice of threshold. Looking at the tables in the appendix, I do still think the detector developed by the author is comparable/is ahead of the baselines (although the margins become a lot closer in several cases), however, I think unless average precision (AP) or AUROC is used as the fundamental metric for all of the experiments, it could promote incorrect conclusions.
- For the experiments with post-processing (JPEG, Blur etc), it would be nice if the authors can clarify as to which of these augmentations are seen during train time and which ones are not. Once again, I think all these experiments should have AP or AUROC reported.
Writing Level Weaknesses
- Line 053 should come when the term residual is used for the first time. It is not intuitive to researchers as to what this term means.
- Equation-2: should be defined properly when introduced.
- Line 112-113 is vague, what does "positive" evidence even mean.
Questions for authors
- Fig 4 shows a "double ascent" kind of phenomenon where the scores shoot up again, at almost twice the so-called optimal patch size, why does this happen?
- From Table 2, residual scaling seems to provide a huge jump on detecting GAN generated images (almost like 15 points), is there a particular reason or any intuition that the authors could provide regarding this?
4 comments
Response to Reviewer Scfh (2/2)
W2 (continued). Calibrated thresholds
To measure how much the choice of threshold affects the comparison, we also calibrated every detector under four protocols (new Appendix D.7):
- (i) one common validation set of held-out SA-1B photographs and their SD 2.1 reconstructions;
- (ii) data close to the training set of each detector: COCO and LSUN photographs with LDM images for AlignedForensics, Stay-Positive and DEAR-r; COCO photographs with SD 2 reconstructions for DDA, and for B-Free and PROBE, whose training data are not available; GenImage SD 1.4 for GenSign;
- (iii) one threshold for all 67 test sets, chosen on a disjoint half of the test images;
- (iv) a separate threshold for each generator, chosen on its own test set, an upper bound that is not available in deployment.
| Recent 11 bAcc | 0.5 | (i) | (ii) | (iii) | (iv) |
|---|---|---|---|---|---|
| Ours (DINOv2-B/7) | .949 | .936 | .936 | .950 | .956 |
| DEAR-r (R50) | .896 | .814 | .898 | .888 | .919 |
| PROBE (DINOv2-L/14) | .853 | .810 | .848 | .872 | .905 |
| DDA (DINOv2-L/14) | .827 | .843 | .837 | .843 | .858 |
| B-Free (DINOv2-B/14) | .772 | .749 | .777 | .780 | .798 |
| AlignedForensics (R50) | .747 | .708 | .824 | .818 | .871 |
| Stay-Positive (R50) | .709 | .677 | .839 | .881 | .909 |
| GenSign (CLIP-L/14) | .643 | .652 | .651 | .653 | .679 |
AlignedForensics and Stay-Positive gain the most from calibration, because at 0.5 the scores of many recent generated images fall below the threshold (W2 above). However, our detector remains first under every protocol, and its default threshold of 0.5 is already close to its per-generator optimum (.949 against .956).
W3. Post-processing: seen augmentations and AUROC
Thank you for the suggestion. Table 18 now marks the post-processing that each detector sees in training (J: JPEG, B: blur, R: downscaling), and Table 19 gives the AUROC of every condition. The table below gives the means over the conditions of each group (200 pairs per recent generator):
| bAcc / AUROC | Seen | JPEG (4) | Blur (3) | Downscale (2) | All 9 |
|---|---|---|---|---|---|
| AlignedForensics (R50) | J B | .647 / .848 | .706 / .904 | .552 / .797 | .646 / .855 |
| Stay-Positive (R50) | J B | .618 / .904 | .644 / .919 | .539 / .798 | .609 / .885 |
| B-Free (DINOv2-B/14) | J B R | .735 / .756 | .734 / .764 | .616 / .657 | .708 / .737 |
| DDA (DINOv2-L/14) | J B R | .745 / .816 | .825 / .896 | .728 / .797 | .768 / .838 |
| PROBE (DINOv2-L/14) | ? | .790 / .890 | .799 / .898 | .641 / .762 | .760 / .864 |
| DEAR-r (R50) | J B | .831 / .910 | .869 / .945 | .723 / .806 | .819 / .899 |
| GenSign (CLIP-L/14) | – | .500 / .460 | .611 / .663 | .590 / .620 | .557 / .563 |
| Ours (DINOv2-B/7) | J R | .774 / .870 | .893 / .959 | .764 / .867 | .811 / .899 |
| Ours (DINOv2-L/7, one seed) | J R | .844 / .940 | .952 / .984 | .859 / .961 | .884 / .959 |
Our detector sees JPEG and downscaling in training but not blur. With DINOv2-B/7, our detector is the most accurate under blur and downscaling and matches DEAR-r in AUROC over the nine conditions, but trails DEAR-r and PROBE under JPEG. With DINOv2-L/7, the backbone size of DDA and PROBE, our detector is the most accurate in every group. We discuss JPEG in General Response §2 and in the reply to 5n62.
Writing
Thank you for these suggestions. We now define the residual where the term first appears in the abstract and the Introduction, and define with Equation 2. We also replace the vague "positive evidence" with what Stay-Positive does: Stay-Positive constrains the weights of the last layer to be non-negative, so that the score rises only with features that indicate a generated image.
Q2. Intuition for the gain on GAN sets
Thank you for this question. Our intuition is that GAN images carry no autoencoder residual, but the last layers of GAN generators are learned upsamplers that also leave fine high-frequency detail, with a magnitude different from the SD 2.1 residual. A detector trained on one magnitude misses this detail, whereas residual scaling trains the detector to respond across magnitudes. Our results support this intuition. The attenuated part of the family (0 < β < 1) helps most: training with this range alone raises the bAcc on the seven GAN sets of Table 3 (the submission's Table 2) from .509 to .655, more than negative β or β > 1 alone, and the full family reaches .700 (three seeds).
We thank the reviewer again, and we would be glad to clarify any point further.
Reviewer Response
I would like to thank the authors for taking the time to perform the requested experiments. This gives me confidence that the half-patch size is helpful as well as the results not being inflated due to the reporting of threshold-less metrics. Furthermore, I also feel confident regarding the robustness to post-processing. I thereby keep my score and recommend acceptance to the conference.
Thank you
We thank the reviewer for showing us that readers also want to see threshold-free metrics such as AUROC. We are also grateful for the detailed comments on the writing, which made the paper clearer. We sincerely appreciate the careful review.
Response to Reviewer Scfh (1/2)
We thank the reviewer for the careful reading, for recognizing the augmentation and the breadth of the evaluation, and for the detailed suggestions.
Thank you for these questions. We would like to clarify that the pretrained patch is 14 px for DINOv2 and CLIP and 16 px for SigLIP2, so the 12 px point in Figure 4 is a resampled size, not the baseline. We repeated every probe with three seeds and added AUROC (Figure 4b). At block 6 of DINOv2, the 7 px and 12 px probes are within the seed spread (.788 ± .014 against .782 ± .010). However, on all three backbones, half the pretrained size exceeds the pretrained size by more than the spread (by .036 to .127 bAcc).
Fine-tuning separates the sizes more clearly. We fine-tuned three ViT-B backbones at five patch sizes with three seeds each, using the standard recipe without residual scaling (new Table 1):
Half the pretrained size is best on every backbone (standard deviation over seeds .001 to .006), and also on a validation split of the training data that contains no test image.
The second rise comes from two factors. First, a smaller patch covers fewer pixels per token, so the fine residual is averaged less; this effect alone favors every smaller size. Second, the resampled filters reproduce the pretrained tokens only at some sizes. We measure this agreement on DINOv2-B with 400 photographs. We compare the tokens of the pretrained 14 px filters on a 448 px crop with those of the filters resampled to k px on the same crop resized to 32k px; both grids are 32×32, and we report the cosine of matching tokens (Table 13). The agreement is high at 14, 12, 10 and 7 px (at least .87), but low at 8 px (.51) and below .3 at 6 px and smaller. Half the pretrained size is therefore the smallest size with high agreement. Sizes just above half break the agreement, and their accuracy falls below that of the pretrained size even after fine-tuning (table above). The score then rises again toward 12 and 14 px, where the agreement returns.
Thank you for asking for threshold-free metrics. Our detector and the reconstruction-trained baselines learn from reconstructions of real photographs and aim to generalize to generators absent from training. This is the deployment setting: a new generator appears, and no labeled images of this generator exist to tune a threshold. We therefore fixed the threshold at 0.5, as a deployed detector would, and reported bAcc as the main metric. The baselines do the same: AlignedForensics reports accuracy at 0.5, DDA reports bAcc without threshold adjustment, Stay-Positive reports accuracy at 0.5 next to AP, and B-Free reports AUC and bAcc at 0.5, noting that, in a realistic setting, calibration images of the tested generator cannot be guaranteed.
Nevertheless, AUROC, AP and TPR at 1% FPR give a complementary view, so we added these metrics to the revision: TPR@1% in Table 2 (the submission's Table 1), AUROC and TPR@1% in Tables 1 and 4 and Figure 4, and all three for the ablation in Table 21. The sweep (Figure 1) keeps one fixed threshold by design, because the sweep tests whether a single threshold remains valid across residual magnitudes. The sign test of the sweep already uses AUROC (reversed residual at β = −0.5 against the photograph). On the added metrics, our detector leads on the recent generators:
A gap between bAcc and AUROC, as for AlignedForensics and Stay-Positive, means that a detector ranks the images well, but the scores of many images from new generators fall below 0.5. At 0.5, these two detectors flag almost no photographs but detect less than half of the recent generated images (.49 and .42). Even without a threshold, the AUROC of these two detectors remains below ours, and the margin is larger at a low false-positive rate (TPR@1% .888 against at most .719 for any baseline).