acceptodds
Under review as a conference paper at ICLR 2027

Most of the Ablation Gain Was the Baseline's Training Budget: A Placebo-Controlled Audit of Evidence-Guided Deepfake Detection

Abstract

Evidence-guided detectors of AI-generated images fuse a forensic evidence map into an image backbone, and the ablation row "without the evidence branch" is read as the contribution of the map. That row is the sum of three terms: how far the baseline was trained, what the fusion module adds without any map, and what the content of the map adds. We measure each term separately. On a frozen cross-generator test set of 22,999 images and five training seeds, our detector beats a linear probe capped at 100 epochs by 18.35 points of true-positive rate at 5% false-positive rate. Training the probe to its own stopping criterion removes 15.77 of those points; a placebo arm, the same module retrained on all-zero maps, accounts for 1.98 more; the content of the map adds 0.60 (two-level 95% interval [−0.96, +2.20]). Against the converged probe the fused detector shows no gain distinguishable from zero on clean images and is 0.033 AUC lower under blur; for the module retrained without a map, 0.029 AUC lower, a probe on the module's own input attributes half of that loss, and nearly all of the corresponding TPR loss, to the module's parameters. The map share is the one quantity that does not move: across four placebos, three training budgets and two resampling schemes the upper end of its macro-AUC interval is at most +0.0103, and every reported map-share interval that excludes zero does so on the positive side. Two of five seeds nevertheless read the map, and those are the two seeds that lose most under blur. For a published method, LaRE², the same ablation read under three seeds and two checkpoint rules spans −0.016 to +0.060 AUC and changes sign for two of three seeds. We define the ablation resolution of a protocol and show that a single run cannot separate these terms. Our evidence covers one detector we built and one published method; it shows that the problem exists, not how common it is.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
ablation study, placebo ablation, placebo control, model evaluation, evaluation methodology, reproducibility, seed variance, AI-generated image detection, deepfake detection, evidence-guided fusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DNXcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper audits the attribution of performance gains in evidence-guided image detectors. It separates baseline training budget, fusion-module effects, and evidence-map content using converged probes, retrained placebo controls, and repeated runs. Experiments cover a custom detector and LaRE², with an additional proposal to quantify the “ablation resolution” of an evaluation protocol.

Critical strengths
  • The paper addresses a relevant experimental-design problem: removing an evidence branch changes both its information and the model architecture.
  • Retrained placebo controls, multiple seeds, and checkpoint comparisons provide useful diagnostics. The distinction between using an evidence map and benefiting from it after retraining is particularly valuable.
  • The authors appropriately acknowledge the limited scope and distinguish small estimated effects from evidence of no effect.
Critical weaknesses
  1. The empirical contribution remains narrow relative to the limited methodological novelty. The headline finding—that baseline training explains most of the apparent gain—is demonstrated on the authors’ custom detector. The LaRE² experiment establishes seed/checkpoint sensitivity under a modified protocol, without reproducing the published ablation or performing the same budget decomposition. Although these limitations are acknowledged, the additional knowledge beyond established evaluation practices is not yet sufficiently compelling.
  2. The LaRE² execution protocol requires clarification. The manuscript states that the released code’s default validation and test lists coincide. While the Python argument defaults coincide, the recommended launch script overrides the test argument, causing the training code to evaluate eight generator subsets. Moreover, learning-rate scheduling depends on test_score, so this distinction can affect training trajectories. The exact code revision and launch arguments are necessary to assess the audit.
  3. The quantitative “ablation resolution” claims need stronger statistical validation. They rely on variance components estimated from only three seeds and two checkpoint rules. From Table 4, the across-seed standard deviations are approximately 0.0385 for best-validation and 0.0071 for latest, questioning the suitability of a shared variance model for a fixed rule. Uncertainty in the estimated resolution and coverage calibration are missing. Additionally, an 80%-power minimum detectable effect should not be interpreted as a hard boundary below which effect direction cannot be inferred.
Questions for authors
  1. What exact LaRE² commit, launch arguments, and learning-rate schedule were used? Why do validation-best and test-best checkpoints coincide under this configuration?
  2. Can the authors report rule-specific seed-level intervals and quantify uncertainty in the resolution estimates, including coverage validation?
  3. Within the existing LaRE² experiment, does controlling baseline training budget reproduce the paper’s principal finding?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DNXc

Q1. Commit, launch arguments, LR schedule; why validation-best and test-best coincide.

The eight-generator test list was not used. In every training log --test_file and --val_file are the same 2,500-image validation list, so the scheduler stepped on validation AUC. We did not use the released launcher, so test_score in the scheduler is validation AUC and no test generator entered the training trajectory.

Table I. LaRE² protocol as recorded in the logs (App. F.1).

ItemValue
Code revisionf8b2966449682845cfbf1bb4ccbb0dd599a686ed
Arguments--model CLipClassifierWMapV6 --clip_type RN50 --epoches 15 --lr 1e-4 --batch_size 48 --test_batch_size 40 --data_size 224 --val_method con --val_ratio 0 --no_strong_aug
--val_file, --test_fileboth val_2.5k.txt
LR scheduleReduceLROnPlateau(mode='max', factor=0.5, patience=4) on validation AUC
Validation-best and test-bestselected on the same list; saved at the same epoch in every run

We have revised Sec 5.1 to state this configuration.

Q2. Rule-specific seed-level intervals, uncertainty in ρ\rho, coverage.

Done, following the reviewer's split by rule (GR-2). Five seeds: best-validation +0.0147,[−0.0220,+0.0515]+0.0147\\,[-0.0220,+0.0515], ρ(1)=0.083\rho(1)=0.083; latest −0.0018,[−0.0155,+0.0120]-0.0018\\,[-0.0155,+0.0120], ρ(1)=0.031\rho(1)=0.031. Intervals on ρ(1)\rho(1) are in GR-2, and the seed-level interval covers about 94% in simulation (Table C). Every gap is below ρ(1)\rho(1) of its own rule.

W3. Shared-variance model; ρ\rho as a boundary. We have estimated the variance separately for each rule from five seeds (Table B), and both rules give the same conclusion. ρ\rho is the minimum detectable effect at power 0.8 (L241–242). The sign statement rests on the data: the sign changes with the rule in three of five seeds.

Q3. Does controlling the baseline budget on LaRE² reproduce the principal finding?

Yes. The budget term is 71% of the standard row (+0.0233+0.0233 of +0.0327+0.0327 AUC; 70% under best-validation), and the map term is −0.0018,[−0.0155,+0.0120]-0.0018\\,[-0.0155,+0.0120] (Table A).

W1. Contribution beyond established evaluation practice.

Established practice controls the headline score; this paper controls the ablation row, and what it finds there was not known.

Prior workWhat it variesWhat it tests
Henderson et al.; Bouthillier et al.; Agarwal et al.seeds, tuning budgetthe headline comparison
Gulrajani & Lopez-Pazmodel-selection rulethe headline comparison
Hooker et al.; Adebayo et al.input removed, model retrainedan explanation method
This paperbaseline budget, retrained placebo, seeds, checkpoint rulethe attribution made by an ablation row

What a reader learns that these do not provide:

  1. On a published method, the row that credits the evidence map reports +0.033+0.033 AUC while the map's measured share is −0.002-0.002; 71% of the row is the baseline's budget (Table A).
  2. The reading of that row changes sign with the stored checkpoint in three of five seeds (Table B2).
  3. The same control confirms a share of +0.10+0.10 on a second published method (Table J), so it separates detectors whose evidence matters from those where it does not.

The reviewer's Strengths name the distinction between using a map and benefiting from it as particularly valuable; items 1–3 are that distinction measured on two published detectors.

Dear Authors,

Thank you for the additional experiments and clarifications. The full decomposition on LaRE², expanded seed analyses, and protocol clarifications strengthen the empirical evidence and address several of my concerns.

However, my concern about the breadth of the empirical evidence and its relevance to contemporary detectors remains. Budget dominance is demonstrated only for the authors’ detector and LaRE² under the adopted protocols, within one benchmark family. The AIDE experiment broadens method coverage, but does not include a training-budget decomposition. I therefore remain unconvinced that the current evidence sufficiently establishes the broader practical significance of the findings for contemporary image detection research.

I will maintain my rating of 2 (Weak rejection).

AuthorsOpenReview ↗

Response to Reviewer DNXc: training-budget analysis on AIDE

Thank you for reading our response closely and for the follow-up. Your point about the missing budget analysis on AIDE was a fair one, and addressing it has made the paper better.

We have run the budget analysis the reviewer asks for on AIDE. Further training of the placebo does not close the observed gap: its AUC changes by −0.0035-0.0035 (seed-level 95% [−0.0111,+0.0041][-0.0111,+0.0041]), and +0.1044+0.1044 remains.

Setting. We continued the ZERO placebo from its 5-epoch checkpoint, with the same augmentation and validation-loss-based stopping and selection (three seeds; two stopped at epochs 18 and 19, and the third is read at its best validation loss within 18 epochs). The gap also did not shrink at a fixed budget of 18 epochs.

We agree with the reviewer that budget dominance is shown for two systems, and AIDE is not a third. The three detectors give three different readings of the same kind of row:

DetectorBudget termGap to the retrained placeboReading
Ours (5 seeds, TPR)+15.77+15.77 pp of +18.35+18.35 (86%)+0.60+0.60 [−0.96,+2.20][-0.96,+2.20]budget
LaRE², CVPR 2024 (5 seeds, AUC)+0.0233+0.0233 of +0.0327+0.0327 (71%)−0.0018-0.0018 [−0.0155,+0.0120][-0.0155,+0.0120]budget
AIDE, ICLR 2025 (3 seeds, AUC)−0.0035-0.0035 [−0.0111,+0.0041][-0.0111,+0.0041]+0.1044+0.1044gap persists

For AIDE the budget term is the change of the placebo under further training.

Why this matters for contemporary detection research. In all three papers the standard ablation row reads the same way: the evidence branch helps. The analysis shows that the LaRE² row is sensitive to budget and checkpoint rule, whereas the AIDE gap is not closed by further training of the placebo. The controls therefore separate a row that the baseline's budget explains from a gap that survives it, which the row alone cannot do, at a cost of one additional run per seed.

Scope. The paper's claim is that the row is a sum of terms that must be measured separately, not that the budget term dominates everywhere. The abstract's last sentence states this, and the camera-ready subtitle will state the scope as well. We have added the AIDE budget analysis to Sec 5.4 of the revision.

Thank you again for your time and for the careful reading of our code and statistics throughout this discussion. The protocol table, the rule-specific analysis and this AIDE experiment are all in the paper because of your comments.

Reviewer_1yMMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what accounts for apparent gains in evidence-guided synthetic image detection. It separates training budget, module/input design, and evidence-content effects using baselines trained under different budgets and stopping criteria, retrained placebos, and real-evidence models. Across five seeds on the authors' detector, extending baseline training accounts for most of the apparent TPR gain, with less from evidence content. LaRE2 results vary with seed and checkpoint selection. The authors also propose a protocol-dependent measure of ablation resolution.

Critical strengths
  1. The controls target attribution in ablation studies. By separating training budget, module/input design, and evidence content, the authors provide a protocol that other researchers can apply to their own detectors.
  2. The main experiments cover several placebos, training budgets, and input controls. The authors report seed instability and negative results, and they keep the conclusions within the scope supported by those results.
Critical weaknesses
  1. The full decomposition is tested mainly on the authors' detector. LaRE2 shows seed and checkpoint sensitivity, but it does not apply the same decomposition to a published method, so the external evidence that the three-way attribution transfers is limited.
  2. I do not find the convergence evidence for LaRE2 sufficient, and the statistical interpretation of ρ, the proposed ablation-resolution estimate, is not adequately supported. The variance model, the assumption that image-sampling errors are shared, and the influence of a single checkpoint all need justification. The different spatial aggregation in R_gs and P also weakens the attribution of P - R_gs to module parameters.
Questions for authors
  1. Could the authors run the full decomposition on LaRE2 or another published evidence-guided method? The most useful comparison would include an original-budget baseline, a baseline trained to a predefined stopping criterion, a retrained placebo with the same architecture, and a real-evidence model. This would show what the same audit reveals about evidence-content gains in a published detector.
  2. Saturating validation accuracy does not by itself establish convergence. I would like to see loss curves for the LaRE2 real-evidence and ZERO runs, results at different training budgets, and an explanation of the seed-0 ZERO outlier. If the authors cannot justify 15 epochs as sufficient, the convergence claim should be qualified.
  3. What variance-component estimators and degrees of freedom are used for ρ? Please also assess the sensitivity of the result to the outlier checkpoint and to the shared image-sampling-error assumption. Because the normal quantile gives a smaller estimate than a t quantile, does "lower bound" refer only to that comparison, or does it also account for uncertainty in the estimated variances?
  4. Could the authors provide seed-level confidence intervals or sensitivity analyses for the small positive JPEG and SHUFFLE differences? I am asking about uncertainty across training seeds, separately from uncertainty from resampling images at a fixed checkpoint, because it bears directly on the "probably positive" claim.
  5. R_gs concatenates the average-pooled spatial features with the global feature, whereas P also reads the unnormalized pre-projection spatial features. Does P - R_gs therefore reflect a difference in input representation and aggregation as well as module parameters? Please control for this difference or qualify the attribution.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1yMM

Q1. The full decomposition on a published method.

Done on LaRE² with the four arms the reviewer lists, five seeds: budget +0.0233+0.0233 (71%), module +0.0112+0.0112 (34%), map −0.0018-0.0018 (−5-5%) of a +0.0327+0.0327 AUC row (Table A). Under best-validation the budget term is again 70% of the row. The three-way attribution transfers. On a second method, AIDE, the same control confirms a map share of about +0.10+0.10 (Table J).

Q2. Loss curves, budgets, the seed-0 ZERO checkpoint.

Per-run training records are in App. F.2 (summary in Table H). Training loss falls from 0.10–0.12 to 0.002–0.024 by epoch 15. The seed-0 ZERO checkpoint is epoch 13, and a same-seed retrain returns the same scores. Each run is read at two budgets, its best-validation epoch (7–14) and epoch 15, and Table B2 reports both for every seed. L428–430 claim convergence on the training distribution only.

Q3. Estimators, degrees of freedom, outlier, shared image error, "lower bound".

Per rule: the sample SD of the per-seed gap (four d.o.f. at five seeds) and the image-bootstrap SD of the paired gap (Table B).

  • "Lower bound" refers to the normal-versus-tt quantile. We have added the variance uncertainty as an interval: ρ(1)∈[0.050,0.24]\rho(1)\in[0.050,0.24] and [0.019,0.089][0.019,0.089].

  • Outlier: without seed 0, ρ(1)=0.038\rho(1)=0.038, above every remaining gap.

  • Shared image error: it changes coverage by about one point (Table C).

Q4. Seed-level intervals for the JPEG and SHUFFLE positives.

Table E gives them at five and ten seeds, from seed-to-seed variation only: JPEG n=5:[−0.0014,+0.0077];n=10:[−0.0016,+0.0083]n=5:[-0.0014,+0.0077];\\ n=10:[-0.0016,+0.0083], SHUFFLE n=5:[−0.0033,+0.0127];n=10:[−0.0011,+0.0067]n=5:[-0.0033,+0.0127];\\ n=10:[-0.0011,+0.0067]. The seed-level intervals include zero, and the point estimates are positive at both sample sizes. The paper's statement is the bound: at ten seeds its upper end tightens from +0.0103+0.0103 to +0.0067+0.0067 (Table D).

Q5. RgsR_{gs} vs PP aggregation.

Qualified: P−RgsP-R_{gs} covers the module's parameters together with its spatial aggregation, and we have relabelled Sec 3.1 and the probe table (Table 6) accordingly. We also trained a linear probe on the unpooled spatial features (RflatR_{flat}). It is a much weaker reference (P−Rflat=+0.25P-R_{flat}=+0.25 clean AUC), so the mean-pooled RgsR_{gs} remains the stronger linear reference on the module's input.

AuthorsOpenReview ↗

To Reviewer 1yMM: a brief update before the discussion closes

Thank you again for your careful questions. Since our response, the comparisons on the second published method have been completed, which extends our answer to Q1 (general comment "Update: AIDE with all three arms at five epochs, and its training-budget analysis"):

  • AIDE, three arms at five epochs: FULL −- ZERO =+0.1009=+0.1009 [+0.0984,+0.1035][+0.0984,+0.1035] and FULL −- SHUFFLE =+0.1005=+0.1005 [+0.0979,+0.1030][+0.0979,+0.1030], with similar gaps under validation selection.
  • Budget on AIDE: further training of ZERO changes its AUC by −0.0039-0.0039 [−0.0133,+0.0054][-0.0133,+0.0054]; a gap of +0.1049+0.1049 remains. All three seeds have met the validation-loss stopping rule.

Together with Table A (LaRE²: budget 71%, map −0.0018-0.0018 [−0.0155,+0.0120][-0.0155,+0.0120]), the audit you asked for in Q1 now covers two published detectors and gives a different reading for each.

Our answers to Q2 to Q5 are unchanged. If anything in them is still unclear, we would be glad to clarify before the deadline, and we would be grateful if you could let us know whether your questions are resolved.

Reviewer_Tc9QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a methodological audit of the "evidence-guided fusion" paradigm in AI-generated image detection.

Core insight: In these papers, the ablation row "remove the evidence branch" is credited to the evidence map. In fact it is the sum of three terms: the baseline's training budget, the gain from the fusion module itself (without looking at the map), and the gain from the evidence map's content. The first two are biases inherent to the comparison, and repeating experiments cannot remove them.

Approach: The ablation row is written as an identity, with a measurement tool for each term:

Train the baseline to its own stopping criterion (measures the budget term). A placebo arm, which keeps the module but replaces the evidence map with a zero, mean, noise or shuffled map and retrains (measures the module term). Full minus placebo (measures the map-content term). These are paired with a two-level bootstrap that resamples both seeds and test images, and small effects are reported only as upper bounds, not as "no effect". The paper also proposes ablation resolution: the smallest component effect a protocol can resolve.

Critical strengths
  1. An exceptionally clean problem decomposition: writing an ablation row as an identity. Section 3.1 explicitly labels µ_F − µ_B100 = G_bud + G_mod + G_map (Eq. 1) as "an identity, not a result", and further splits the module share into "input representation" and "module parameters" (two probes, R_g and R_gs). This turns a vague community consensus ("ablation rows can't be fully trusted") into three separately measurable quantities. It also shows that the bias G_bud + G_mod is a property of the comparison itself and cannot be removed by more repeated runs. The argument is rigorous, domain-independent and highly transferable.

  2. The control design is more rigorous than comparable work. The placebo is not casual zeroing. It must meet three admission conditions (C1/C2/C3), and four variants are constructed: ZERO, MEAN, NOISE and SHUFFLE. SHUFFLE preserves the marginal distribution of the evidence maps and breaks only the pairing. Crucially, the placebo arm is retrained rather than zeroed at test time (echoing remove-and-retrain). Appendix B reproduces this with forward hooks, weight norms (<2×10⁻⁵) and weight decay, showing that the ZERO arm's map pathway is truly "off". This rules out the objection that the placebo arm still secretly reads the map.

  3. Uncertainty is handled honestly, with "small" explicitly distinguished from "zero". A two-level bootstrap resamples both the 5 seeds and the images within class × generator strata (B=10,000, with thresholds recomputed each time). The authors admit that with 5 seeds the percentile bootstrap is anti-conservative at the seed level, so the headline map share also gets a wider seed-level t interval. Small effects are written up in equivalence/bound form (the TOST tradition, citing Schuirmann and Lakens), reporting an "upper bound of +0.0103 AUC" rather than "no effect". The "What moves and what does not" matrix in Table 3 lays out each conclusion's sensitivity to 8 choices at once, a self-audit that many rebuttal-style papers lack.

  4. Rare preregistration and disclosure of deviations. Appendix H lists every deviation from the preregistered text with reasons, including ones clearly unfavorable to the authors: Three conclusions from the original three seeds did not hold up at five seeds: the clean map share flipped sign (−0.0004→+0.0010), the blur-stability advantage fell from 8.4× to 1.8×, and the blur gap went from −0.015 to −0.002. The results-freeze date was pushed back. Five planned analyses were abandoned. Appendix G also preregisters future experiments together with the sentence to be written for each possible outcome. This is close to a model of open-science practice.

Critical weaknesses
  1. The universal tone of the title and abstract vs. a very narrow evidence base. The title, "Most of the Ablation Gain Was the Baseline's Training Budget", is a field-level assertion, but the evidence is one author-built detector, one published method and one benchmark family, with 5 and 3 seeds. The authors do qualify this in the last sentence of the abstract and in the Limitations ("shows the problem exists, not that it is universal"), but the main text and title spread far beyond what the evidence supports. Worse, the analysis that would best answer "how common is it", a survey of reporting practice, is among the five analyses the authors deliberately dropped (Appendix H). The paper thus stops at an existence proof, when a frequency estimate is what reviewers most want.

  2. The headline number depends on an author-defined baseline, with straw-man risk. The loudest conclusion, "15.77 of the +18.35 pp comes from budget", uses B100 as its denominator, a linear probe capped at 100 epochs by the authors themselves. The authors stress that it was "the first choice before analysis, not a weak baseline built for comparison", and they add a cap-500 control. But B100's stopping criterion never triggered (patience 10 did not rescue it), so the title rests on a clearly under-trained control. The module-share conclusion is also highly sensitive to how the reference probe is stopped. With the L-BFGS optimum, the clean TPR share goes from containing zero to +2.70 [+0.003, +5.22], excluding zero, while the blur TPR share no longer excludes zero (Appendix B). The authors report this honestly, but it shows that the three-term split in Eq. 1 is not invariant: the "shares" depend on the choice of reference.

  3. Statistical power is insufficient to support mechanistic conclusions. With n=5, the smallest possible exact one-sided p for a rank correlation is 1/120, and the authors themselves make no significance claim. The Section 4.3 finding that two seeds read the map and are the most fragile under blur is therefore anecdotal, yet it appears in the abstract and the title-level narrative. Each variance component of LaRE²'s decomposition (σ̂_seed, σ̂_rule, σ̂_int) has only 2 degrees of freedom, and σ̂_int=0.0261 is the largest and directly determines the 0.083 resolution. The paper's most powerful number, the single-run resolution, thus rests on its most fragile estimate. Appendix A also admits the independence assumption fails for seed 0.

  4. The LaRE² audit has substantive comparability gaps and is driven by a single outlier. Section 5 uses a 25,000-image subset of GenImage SD V1.4, an author-set 15-epoch budget (set by a pilot), and two zeroing switches the authors added themselves. The original paper's protocol and test data are not reproduced, and the authors explicitly state they make no comparison with the originally reported ablation. Strictly, then, Section 5 audits the instability of the authors' own LaRE² variant under their protocol, not the conclusions of that CVPR paper, yet Figure 1b and the abstract present it as equivalent evidence. More critically, the −0.016 to +0.060 span comes mainly from a single ZERO-map checkpoint with AUC 0.894 (the other 11 are all 0.93–0.96). This outlier also dominates the interaction variance, so the conclusion that the checkpoint rule can flip the sign may be hostage to one point.

  5. The positive output is thin, and the presentation sacrifices readability. The only constructive content is the four practices in Section 6 (converged baselines, placebo arms, input alignment, pre-fixing seed sets and checkpoint rules) plus a cost estimate of one extra training run per seed. There is no tooling: no ready-to-use audit script or checklist, no community-usable packaging of the resolution formula, and no actual re-audit of existing detectors. The main text is extremely dense with numbers, while key content (the L-BFGS reversal, per-generator intervals, the swap protocol, preregistration deviations) is pushed to appendices. The abstract is crammed with seven or eight confidence intervals, and Table 3's four-level notation (sign/size/±0/stays) requires repeated look-ups. For readers outside this subfield, the reading cost clearly exceeds the argumentative payoff. The SHUFFLE arm is also mildly circular. With SHUFFLE as the reference, the map share is the only positive interval that excludes zero, [+0.0003, +0.0103], yet the authors say part of it is the SHUFFLE arm's own penalty from mismatched input pairs. The paper does not separate whether this "positive effect" belongs to map content or to a placebo defect.

Questions for authors

no

2 comments
AuthorsOpenReview ↗

Response to Reviewer Tc9Q

W1. Title and evidence base.

The evidence is widened. The reviewer describes the decomposition as domain-independent and highly transferable; the new results are the first transfer: the same three instruments, unchanged, on LaRE² and AIDE. Ten seeds for our detector; LaRE² with five seeds and the full decomposition (Table A); and a second published method, AIDE (Table J). How often each term dominates can only be measured by re-running each method; the two published methods above are that measurement, and they show that the answer differs by method. Title: we agree that it should state its scope. The author guidelines state that no title change is permitted until the end of the discussion period, so the camera-ready subtitle will read "A Placebo-Controlled Audit of One Detector and Two Published Methods". The main title's statement holds in both systems where the baseline's budget was controlled (86% of the row for our detector, 71% for LaRE²), and the abstract's last sentence and Sec 7 (L472–475) limit the claim to existence and to one benchmark family.

W2. B100B_{100}; dependence on the reference.

The budget term is not specific to B100B_{100}: it is +0.040+0.040 AUC against the cap-500 probe (App. B), and 71% of the row on LaRE² with that method's own baseline. B100B_{100} was the baseline before the analysis (L259–261); its criterion fires only at epochs 504–622 (L783–784), and that distance is what the budget term measures. The map share is independent of the baseline by construction (L295–296), and the module share's AUC conclusions hold under the L-BFGS reference as well; the TPR sensitivity is the "Probe stopping" column of Table 3.

W3. Statistical power.

The recorded prediction holds at n=10n=10: ρs=−0.66\rho_s=-0.66, exact one-sided p=0.022p=0.022 (Table D). For LaRE², the variance is re-estimated with five seeds per rule, with four degrees of freedom and no independence assumption; the single-run resolution is 0.083, the paper's value (Table B).

W4. LaRE² audit; single outlier.

We have added the full decomposition to Sec 5, and its conclusions hold without the 0.894 checkpoint.

  • Decomposition: budget 71%, module 34%, map −5-5% (Table A).

  • Sign: the gap changes sign with the rule in seeds 42, 0 and 2; only seed 0 involves that checkpoint (Table B2).

  • Resolution: without seed 0 it is 0.038, above every remaining gap.

  • The checkpoint: a same-seed retrain returns identical scores.

  • Protocol: the budget was fixed before any cross-generator evaluation and is the same for every arm (App. F). Sec 5.2 states the scope: the released code under our protocol, with no comparison to the originally reported ablation (L455–458).

W5. Positive output, readability, SHUFFLE.

  • Re-audit of existing detectors: LaRE² and AIDE (Tables A, J). Tooling: we have added audit_ablation.py and a checklist to the supplementary material.

  • Readability: done as the reviewer asks. We have shortened the abstract to its headline numbers; we have summarised the L-BFGS result, the per-generator intervals, the swap protocol and the pre-registration deviations in the main text; and we have added a legend table to Table 3.

  • SHUFFLE: L313–316 separate the two: the SHUFFLE arm scores 0.9358 with swapped or correctly paired maps, so its deficit is the placebo's own penalty. ZERO, MEAN and NOISE agree within 0.0008 (Table 2a); +0.0103+0.0103 is the most conservative of the four bounds.

AuthorsOpenReview ↗

To Reviewer Tc9Q: a brief update before the discussion closes

Thank you again for your detailed review. Since our response, two results have been added that bear on your W1 and W5 (general comment "Update: AIDE with all three arms at five epochs, and its training-budget analysis"):

  • A second published method, completed comparisons. On AIDE, two retrained placebos (ZERO and SHUFFLE) leave a gap of about +0.10+0.10 AUC at five epochs, under both checkpoint rules, and further training of ZERO does not close it.
  • Three detectors, two different readings. Budget explains most of the row for our detector and LaRE²; on AIDE the gap persists. This is the scope the camera-ready subtitle will state.

For your other points, in one line each:

  • W2: the budget term holds against the cap-500 probe and on LaRE² with its own baseline (71%).
  • W3: the recorded prediction holds at ten seeds (ρs=−0.66\rho_s=-0.66, exact p=0.022p=0.022).
  • W4: the full decomposition is in Sec 5.3, and its conclusions hold without the 0.894 checkpoint.
  • W5: we have shortened the abstract, moved the appendix results you named into the main text, and added a legend to Table 3.

If any of W1 to W5 remains open, we would be glad to address it before the deadline. If they are resolved, we would be grateful if you could reflect that in your final assessment.

Comments on the paper

3 comments
Area_Chair_oKjBOpenReview ↗

Reviewer's responses to authors

Dear reviewers,

If not done already (I have seen one reviewer replying already - thank you rev. DNXc ), could you please take a look and reply briefly to the "Official Comment(s) by Authors" / Rebuttals, possibly by an acknowledgement that you have seen it.

Feel free to respond in more detail, in particular, if you have further questions or opinions to the replies from the authors.

With best regards, an AC

AuthorsOpenReview ↗

Update: AIDE with all three arms at five epochs, and its training-budget analysis

We thank the reviewers and the Area Chair for the discussion so far. Two results on the second published method, AIDE, are now complete, and both are in Sec 5.4 of the uploaded revision.

1. All three arms at five epochs (replaces Table J, where the three arms were compared at one epoch).

Arm (three seeds, macro AUC, frozen test set)Last checkpointCheckpoint selected on validation accuracy
FULL0.94060.9415
ZERO placebo0.83970.8400
SHUFFLE placebo0.84020.8402
FULL −- ZERO+0.1009+0.1009 [+0.0984,+0.1035][+0.0984,+0.1035]+0.1015+0.1015 [+0.0982,+0.1047][+0.0982,+0.1047]
FULL −- SHUFFLE+0.1005+0.1005 [+0.0979,+0.1030][+0.0979,+0.1030]+0.1013+0.1013 [+0.0978,+0.1048][+0.0978,+0.1048]

Intervals are paired seed-level 95% intervals. Both placebos leave a gap of about +0.10+0.10 AUC under both checkpoint rules.

2. Training-budget analysis (requested by Reviewer DNXc). Training the ZERO placebo further, until its validation-loss stopping rule is met, changes its AUC by −0.0039-0.0039 [−0.0133,+0.0054][-0.0133,+0.0054]; a gap of +0.1049+0.1049 [+0.0955,+0.1142][+0.0955,+0.1142] remains. All three seeds met the stopping rule at epochs 18, 29 and 19, respectively; their validation-loss-selected checkpoints are epochs 8, 19 and 9 (seeds 42, 0 and 1).

What the three detectors show together. The standard ablation row reads the same way in all three papers: the evidence branch helps. Measured term by term, the baseline's budget explains most of the row for our detector (86%) and for LaRE² (71%, with a map term whose interval includes zero). On AIDE, both retrained placebos leave a gap under both checkpoint rules, and further training of ZERO does not close it. The row alone cannot tell these cases apart; the controls can, at a cost of one additional run per seed.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers and the Area Chair. For convenience, this is where each concern stands.

ConcernRaised byStatus
LaRE² execution protocolDNXcClarified with training logs (Table I). DNXc acknowledged that the additional experiments and clarifications strengthen the empirical evidence and address several concerns
Three-term decomposition on a published methodall threeDone on LaRE² with five seeds: budget 71% of the row (Table A)
Statistics of the ablation resolutionall threeRe-estimated per checkpoint rule with five seeds, with intervals and coverage (Tables B, B2, C)
Budget analysis on AIDEDNXc (follow-up)Done: all three seeds met the stopping rule; further training of ZERO changes its AUC by −0.0039-0.0039 [−0.0133,+0.0054][-0.0133,+0.0054], and a gap of +0.1049+0.1049 remains
Evidence base and scopeTc9Q, DNXcTen seeds on our detector; two published methods; the camera-ready subtitle will state the scope
Readability and toolingall threeAbstract shortened, appendix results summarised in the main text, legend for Table 3, audit script

What is unchanged. The central question and conclusions of the submission. The revision adds the requested experiments and clarifies the presentation.

What remains. DNXc's remaining concern is the breadth and broader practical significance of the evidence. The completed AIDE analysis adds a budget test on a second published method; evaluation beyond one benchmark family remains future work, as stated in Sec 7. The paper's claim is that an ablation row is a sum of terms that must be measured separately. The three detectors now show two different readings: the baseline's budget explains most of the row for our detector and LaRE², while the AIDE gap survives further training of the placebo.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.