acceptodds
Under review as a conference paper at ICLR 2027

Ranking Is Not Enough: Shared-Threshold Robustness in AI-Generated Image Detection

Abstract

AI-generated image detectors are often deployed on images that have been compressed, resized, blurred, transmitted, or re-digitized. These transformations can shift detector scores even when their within-condition ranking remains informative. As a result, strong conditionwise detection does not guarantee that one fixed orientation and threshold will work across all deployment conditions. We formalize this distinction in two stages. First, we study whether detector evidence itself survives processing. Our scalar response, , compares the image change caused by one application of a fixed reconstruction operator with the additional change caused by a second application. We derive a response-survival decomposition that tracks how clean real-versus-generated separation is retained or lost through processing distortion and reconstruction-processing interaction. Second, we characterize when all conditions admit one common decision rule. Exact threshold intersections determine shared-threshold feasibility, while total variation gives lower bounds on any decision restricted to the response. Empirically, we evaluate across two real-image sources, six generator families, and 32 processing conditions using independent calibration and confirmation. We then apply the same protocol to 58 publicly available detectors. The results show both regimes: some scores support a useful shared operating point, while many retain strong conditionwise ranking but fail under one fixed rule. For , the best shared threshold remains close to chance in its worst condition. These findings separate response survival from threshold compatibility and show why both are necessary for robust detector deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI-generated image detection, shared-threshold robustness, response survival

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6crjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether AI-generated image detectors can use one score orientation and threshold across different sources, generators, and processing conditions. It develops a reconstruction response q1q_1, response-survival analyses, and shared-decision criteria, and evaluates 58 detectors across 384 conditions. The deployment question is relevant, and the audit provides useful evidence. However, the additional theoretical insight, attribution of performance loss, and reproducibility require further support.

Critical strengths
  1. The paper evaluates a practical deployment requirement: maintaining one operating rule across heterogeneous conditions.
  2. Independent calibration and confirmation, exhaustive threshold evaluation, and simultaneous CDF bounds strengthen the shared-rule conclusions.
  3. The matched comparison covers many recent detectors and reports both failures and methods supporting useful shared operating points.
Critical weaknesses
  1. The theoretical analysis does not sufficiently explain the deployment failures. The threshold-intersection criterion largely formalizes feasibility, and the confidence bounds use standard tools. The more specific analysis in Theorem 3.1 and Figure 3 concerns mean separation, but its connection to the observed shared-threshold incompatibility is not demonstrated.

  2. The detector comparison lacks technical interpretation. Figure 5 and Table 10 report performance differences without linking them to training procedures, representations, augmentations, or score construction. Local discrimination loss is also not separated from direction and threshold incompatibility. A focused technical analysis is needed to explain why representative methods succeed or fail.

  3. Essential implementation details are missing. Appendix C.1 does not identify the model and checkpoint implementing AA, the image-sizing rules, or their order relative to processing TT. Appendix C.2 provides a baseline-checkpoint selection policy but lacks identifiers that uniquely specify the evaluated checkpoints.

Questions for authors
  1. Could the authors provide a concrete example showing how the response-survival analysis explains an observed shared-rule failure, and clarify which conclusions are restricted to mean separation?
  2. Could the authors use existing scores to compare representative strong and weak detectors, separate local discrimination loss from direction and threshold incompatibility, and relate these patterns to specific design choices?
  3. Could the authors specify the implementation and checkpoint of AA, the image-sizing and processing order, and the checkpoint identifiers used for the public detectors?
Reviewer_jPvaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the limitations of AUROC-based evaluation in AI-generated image detection, emphasizing that strong ranking performance does not necessarily guarantee reliable classification under a shared decision threshold. The authors develop a theoretical framework for response survival and shared-threshold feasibility, supported by experiments across 384 evaluation conditions and 58 existing detectors. The results reveal substantial worst-case classification errors despite relatively strong ranking performance, highlighting the importance of fixed-threshold robustness evaluation.

Critical strengths

S1. A practically relevant evaluation lens, formalized as an exactly computable metric. The paper makes explicit that per-condition ranking and a single deployable operating point are distinct properties. It turns the latter into a concrete quantity, the critical shared class error E*, computed with atom-aware tie handling and exhaustive enumeration of all distinct thresholds in both orientations, so the result does not depend on a threshold grid. The underlying observation is not new, but a standardized fixed-rule metric reported alongside AUROC would be a useful addition to detector evaluation practice.

S2. A large, matched benchmark of public detectors. All 58 detectors are scored on the same 8,192 parent images under the same 32 processing conditions (about 15.2M scores). Checkpoints are fixed in advance, and rule selection (calibration) is separated from evaluation (confirmation). Table 10 is a valuable resource in its own right, independent of the q1 analysis.

S3. Transparent reporting. The paper discloses full per-cell estimates (Tables 7–9) and reports negative results for its own response q1 rather than omitting them. This makes the analysis independently checkable.

Critical weaknesses

W1. The central claim is not tested. Fig. 5 measures ranking with orientation-free AUROC* = max{AUROC, 1−AUROC} averaged over cells. This grants exactly the per-cell orientation freedom a shared rule lacks, while E* is determined by the single worst cell. Without the per-cell oracle error max_c min_{σ,τ} max(FPR_c, FNR_c) or the paper’s own TV-based Bayes risk (Theorem 3.3), which Table 5 reports as “Unavailable”/”Inconclusive,” threshold incompatibility cannot be distinguished from near-chance performance in the worst cell. For q1, Appendix D.3 shows that SD 1.4 and Wukong are negative in all 32 conditions including clean, and no processing changes any cell’s sign. The failure is therefore generator-driven, not processing-driven as Fig. 1(a) suggests.

W2. Headline statistics are artifacts of the inference construction. (a) The calibration-selected rule is the constant all-real decision, whose FPR = 0 and FNR = 1 are exact in every cell. Yet it is assigned the intervals [0, 0.087347] and [0.912653, 1]. The Table 1 lower bound 0.456326 and the 4.37 pp “advantage ceiling” are therefore exactly 0.5 − ε/2 and depend only on n and m; Fig. 2 nevertheless labels this value “Exact full class minimum.” (b) Every interval in Tables 7–9 has half-width 0.34939 = 4ε_DKW, determined by the a priori range of q1, while point estimates are 10⁻³–10⁻⁵. “All intervals contain zero” is thus uninformative, and reading it as “substantial sign heterogeneity” (C.3, D.6) is incorrect.

W3. The survival decomposition is tautological, and its main conclusion is an aggregation artifact. Eq. (2) holds for any g ≥ 0 by construction, and the selection criterion for g is unstated, so the “12.7% retained / 99% consumed” split is not identified. The clean margin is an equal mixture of 12 pairs, four of them negative. Under Gaussian 6, the eight positive cells sum to about +7.1×10⁻⁴ and the four negative cells to about −7.2×10⁻⁴, while every cell keeps its sign. “Blur nearly erases the margin” thus reflects cancellation, contradicting Appendix D.6’s own warning. No certificate L_T is evaluated on any cell.

W4. Reproducibility and design. The reconstruction operator A is never specified (architecture, checkpoint, resolution). The real sources are unidentified. The 384 cells are a Cartesian product of 2 real and 6 generated sets without content matching. Since the q1 worst case is witnessed by a real source under JPEG 35, content- or format-driven location shift is a plausible alternative explanation that the paper’s own control leaves “Inconclusive.”

W5. Limited novelty and disconnected contributions. The AP-versus-fixed-threshold gap was noted as early as Wang et al. (2020), and Theorems 3.2–3.3 restate a definition and a standard TV identity. The 58-detector benchmark does not use the q1 theory. Its main metric saturates (46/58 methods sit exactly at the constant rule), and top methods are indistinguishable under the ±0.10 simultaneous radius. Observation 6 and Table 5 are not interpretable from the main text.

Questions for authors

Q1. Using the existing confirmatory data, could the authors report E* separately for the clean condition alone, for each generator paired with both real sources, for each real source paired with all generators, and for each processing family? Such a breakdown would show whether the shared-threshold failure is induced by processing or is already present across generators before any processing is applied. For q1 in particular, it would be informative to know how large E* is in the clean condition, given that SD 1.4 and Wukong show the opposite sign in every condition.

Q2. The central claim is that failure reflects threshold incompatibility rather than information loss. To support it, could the authors report, for each detector, the per-cell oracle error max_c min_{σ,τ} max(FPR_c, FNR_c) and the worst-cell AUROC under a fixed orientation rather than AUROC*? Ideally these would be accompanied by the TV-based Bayes risk of Theorem 3.3. It would also help to know how much of the gap closes under condition-aware calibration, for example with one threshold per processing family. This is the question whose answer is most likely to change my assessment.

Q3. The calibration-selected constant all-real rule has FPR = 0 and FNR = 1 exactly in every cell, yet it is assigned intervals of width ε. This appears to be what determines the lower bound of 0.456326 in Table 1. Could the authors recompute Table 1 and Fig. 2 treating constant rules exactly, and clarify the label “Exact full class minimum”? Relatedly, every interval in Tables 7–9 has the same half-width of 4ε_DKW. Could the authors provide data-adaptive intervals and explain how they reconcile with the bootstrap intervals in Fig. 3?

Q4. What criterion is used to select g, and how sensitive are the reported “retained” and “consumed” percentages to that choice? Appendix D.6 itself cautions against averaging cells of opposite sign. I would therefore also ask the authors to report the decomposition separately for each source–generator pair and to state whether a positive certificate L_T is obtained for any cell.

Q5. Could the authors specify the reconstruction operator A, including its architecture, checkpoint, input resolution, and preprocessing, as well as the datasets used as real-image sources? I would also like to know whether real and generated images are matched in content and file format, for instance JPEG-compressed real images versus PNG generated images. If they are not, does the location shift of q1 persist under a format-matched comparison?

Q6. How sensitive are the conclusions to the most extreme conditions, for instance when JPEG 35 and 50 and the stronger blur–resize settings are excluded? Since 46 of the 58 methods sit exactly at the constant-rule value, a coverage curve would make the metric considerably more informative. Such a curve would show the fraction of cells that satisfy a given error level at the best shared threshold.

Reviewer_y6iVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper asks whether AI-generated image detectors can use one fixed threshold across different generators and image processing (e.g., blur, resizing, JPEG). It proposes a reconstruction-based score, q1, and a metric, E*, which measures the worst-case error of the best single threshold. Across 384 conditions, q1 performs close to chance: its worst-case balanced error is at least 0.456, and E* is 0.993. Among 58 public detectors, 46 have a shared-threshold balanced error of exactly 0.5 (chance level), even though many still separate real and fake well within each condition.

Critical strengths
  • Large and carefully controlled evaluation. The paper evaluates 58 public detectors on the same 8,192 images under the same 32 processing conditions (about 15 million scores). It uses separate calibration and test data, and reports statistical confidence bounds rather than single numbers. This makes the comparison across detectors fair and reliable.
  • Highlights an overlooked deployment problem. Most papers report accuracy or AUROC averaged over conditions, which can hide failures. This paper shows that a detector can separate real and fake well within each condition but still fail when one threshold must work everywhere (e.g., OmniAID and MDMF). The proposed metric, E*, is simple and directly measures this worst-case failure. This is a useful perspective for evaluating detectors in real deployment.
Critical weaknesses
  • Limited practical utility. The paper shows that 46 of 58 detectors perform at chance level when one fixed threshold is used across different processing and generators (Table 10). Yet some of them still separate real and fake well within each condition. For example, OmniAID and MDMF keep AUROC around 0.74–0.95 but are at chance under a shared threshold. However, the paper does not explain how to fix this. As a result, readers learn that the problem exists, but not how to solve it.
  • Limited practical value of the q1 analysis. A large part of the paper (the response-survival decomposition and its theory) is built around the authors’ own score q1. However, q1 performs close to chance under a shared threshold, and the decomposition cannot be applied to existing detectors, which are black-box networks. The paper’s most useful findings (E* and per-condition AUROC for the 58 public detectors) do not depend on q1 at all. It is therefore unclear what the q1 analysis adds beyond simply applying AUROC and E* to existing detectors. The authors should clarify the practical benefit of the q1 analysis, or show how it could guide improvements to real detectors.
  • Unclear real-image sources. The real images are labelled “AIGC natural” and “AIGI StyleGAN3”, but “AIGC” and “AIGI” mean AI-generated, so these labels are contradictory. “StyleGAN3” is also used as a generator. The paper does not say which datasets these real images come from or what they contain. The authors should cite the datasets, describe their content, and use clear labels.
Questions for authors
  • Real-image sources. Which datasets do the two real sources (“AIGC natural” and “AIGI StyleGAN3”) come from ?
  • Role of q1. What does the q1 analysis add beyond applying AUROC and E* to existing detectors? The main practical findings for the 58 detectors do not seem to depend on q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.