- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether AI-generated images misclassified by a final-layer CLS classifier still contain detectable evidence in intermediate layers and patch tokens. It introduces mechanism-aligned evidence probing (MAEP), which selects mechanism-specific linear probes and traces their responses across a frozen DINOv3 backbone. Based on these observations, the authors propose MIRAGE, a post-training method that reuses the detector’s linear head to score internal patch features. These scores are converted into empirical tail probabilities using real-image calibration data and aggregated to reconsider samples initially classified as real. Experiments report improvements across cross-generator, localized-editing, reconstruction, and degradation benchmarks.
Critical strengths
The paper addresses a concrete limitation of image-level detection: local generation traces may remain detectable even when the final CLS classifier misses them. Connecting internal representation analysis to a practical correction mechanism gives the work a clear motivation.
MIRAGE requires no additional trainable parameters beyond the existing detector. Reusing the same linear head across internal layers makes the correction procedure relatively simple, although the underlying backbone remains large.
The paper includes meaningful controls against final-CLS threshold adjustment and internal CLS-only scoring. The patch-based method outperforms these alternatives, providing evidence that the gains are not explained solely by changing the final classification threshold.
Critical weaknesses
- The Chameleon baseline is inconsistent across experiments. The main paper and Table 13 report DINOv3-Linear at 93.30% real accuracy, 89.54% fake accuracy, and 91.42% balanced accuracy. Table 12(a), however, reports a “matched” clean baseline at 97.34%, 87.57%, and 92.46%, respectively, while the MIRAGE result remains unchanged. Relative to the matched baseline, the balanced-accuracy improvement is 1.82 points rather than 2.86 points, and real accuracy decreases by 5.35 points rather than 1.31 points. These differences materially affect the claimed gain and its cost. The paper needs a consistent checkpoint and evaluation protocol for the central comparison. I think the result is wrong.
2.The calibration procedure establishes an empirical operating point, but its out-of-sample false-positive behavior remains insufficiently characterized. The same candidate-real reference set is used to estimate the marginal tail probabilities and calibrate their minimum. The calibration images also come from the training split used to fit the linear head; a disjoint calibration partition is not established. Matching approximately 1% FPR on these samples therefore does not demonstrate comparable control on unseen real images. At the nominal 1% setting, MIRAGE’s real accuracy is 91.99% on Chameleon and 83.06% on the original RRDataset condition. The authors acknowledge domain shift, but these substantial deviations deserve more emphasis and evaluation.
3.MAEP does not fully establish that its trajectories measure the preservation or disappearance of generative evidence. The probes are transferred across layers and token types, and some are trained on LoRA-adapted features before being applied to the frozen backbone. Changes in their margins may reflect probe–representation compatibility rather than changes in available forensic information. Probe roles are also selected manually through visual inspection, without a reproducible selection criterion. Layer-specific probes, quantitative localization results, and analysis restricted to final-CLS false negatives would strengthen the proposed interpretation.
4.The contribution of joint rarity calibration is not cleanly isolated. Table 13 compares empirical min-p against naive OR and AND rules with substantially different realized calibration FPRs: approximately 1.001%, 2.103%, and 0.376%. Their accuracy differences therefore mix aggregation choices with different operating points. Moreover, thresholding the minimum marginal p-value is itself an OR rule with a jointly calibrated cutoff. The relevant comparison is against alternative aggregation scores calibrated to the same false-positive budget.
5.Important failures remain in the scenarios central to the paper’s motivation. On DDA-COCO, MIRAGE reaches only 24.33% fake accuracy; the FLUX.1 and SD3.5-L splits remain at 5.89% and 4.14%. On DRCT-inpaint, fake accuracy is 37.97%. These are improvements over the baseline, but most fake images remain undetected. The evidence supports partial recovery rather than a broad resolution of reconstruction and localized-editing failures.
6.Generality and practical cost are not established. The main experiments use DINOv3-ViT-7B/16, with no evidence that the same approach transfers reliably to smaller backbones or other representation families. No latency, throughput, or peak-memory comparison quantifies the cost of extracting and scoring intermediate patch features. Appendix F also states that the release excludes trained heads and full data-preparation and feature-extraction pipelines, leaving substantial work required for end-to-end reproduction.
Questions for authors
1.Why do the main Chameleon comparison and Table 12(a) use different DINOv3-Linear results? Which baseline uses exactly the same head, preprocessing, and samples as MIRAGE?
2.Are the real images used for calibration disjoint from those used to train and select the linear head? How do the results change with separate reference and threshold-calibration splits?
3.What statistical interpretation is intended for the nominal 1% tolerance: an empirical calibration-set budget or an out-of-sample FPR target? Can you report uncertainty intervals and results on held-out real domains?
4.Do the MAEP trajectories remain consistent when using independently trained layer-specific probes and a fixed, quantitative probe-selection rule? Can you quantify patch localization specifically on images misclassified by the final CLS detector?
5.How does empirical min-p compare with maximum-margin, mean-score, and other aggregation rules when all are calibrated to the same realized FPR?
6.Which development data determined the layer set, K, and alpha before testing? The paper states that settings were fixed before testing but also highlights that alpha = 1% maximizes the six-benchmark mean; please clarify the selection procedure.
7.What explains the very low fake accuracy on FLUX.1 and SD3.5-L reconstruction samples? Is useful internal evidence absent, or does the current probe and aggregation procedure fail to recover it?
8.Can you provide results on a smaller backbone, measure inference overhead relative to DINOv3-Linear, and release the trained heads and scripts needed to reproduce the main tables?
Update: Main linear-head weights released
We have now released the trained main linear head H0 in the supplementary materials at
supplementary_material/MIRAGE_main_H0_single_head.zip. The package includes the weights, loading instructions, and identity verification information.