- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper audits the attribution of performance gains in evidence-guided image detectors. It separates baseline training budget, fusion-module effects, and evidence-map content using converged probes, retrained placebo controls, and repeated runs. Experiments cover a custom detector and LaRE², with an additional proposal to quantify the “ablation resolution” of an evaluation protocol.
Critical strengths
- The paper addresses a relevant experimental-design problem: removing an evidence branch changes both its information and the model architecture.
- Retrained placebo controls, multiple seeds, and checkpoint comparisons provide useful diagnostics. The distinction between using an evidence map and benefiting from it after retraining is particularly valuable.
- The authors appropriately acknowledge the limited scope and distinguish small estimated effects from evidence of no effect.
Critical weaknesses
- The empirical contribution remains narrow relative to the limited methodological novelty. The headline finding—that baseline training explains most of the apparent gain—is demonstrated on the authors’ custom detector. The LaRE² experiment establishes seed/checkpoint sensitivity under a modified protocol, without reproducing the published ablation or performing the same budget decomposition. Although these limitations are acknowledged, the additional knowledge beyond established evaluation practices is not yet sufficiently compelling.
- The LaRE² execution protocol requires clarification. The manuscript states that the released code’s default validation and test lists coincide. While the Python argument defaults coincide, the recommended launch script overrides the test argument, causing the training code to evaluate eight generator subsets. Moreover, learning-rate scheduling depends on test_score, so this distinction can affect training trajectories. The exact code revision and launch arguments are necessary to assess the audit.
- The quantitative “ablation resolution” claims need stronger statistical validation. They rely on variance components estimated from only three seeds and two checkpoint rules. From Table 4, the across-seed standard deviations are approximately 0.0385 for best-validation and 0.0071 for latest, questioning the suitability of a shared variance model for a fixed rule. Uncertainty in the estimated resolution and coverage calibration are missing. Additionally, an 80%-power minimum detectable effect should not be interpreted as a hard boundary below which effect direction cannot be inferred.
Questions for authors
- What exact LaRE² commit, launch arguments, and learning-rate schedule were used? Why do validation-best and test-best checkpoints coincide under this configuration?
- Can the authors report rule-specific seed-level intervals and quantify uncertainty in the resolution estimates, including coverage validation?
- Within the existing LaRE² experiment, does controlling baseline training budget reproduce the paper’s principal finding?
3 comments
Dear Authors,
Thank you for the additional experiments and clarifications. The full decomposition on LaRE², expanded seed analyses, and protocol clarifications strengthen the empirical evidence and address several of my concerns.
However, my concern about the breadth of the empirical evidence and its relevance to contemporary detectors remains. Budget dominance is demonstrated only for the authors’ detector and LaRE² under the adopted protocols, within one benchmark family. The AIDE experiment broadens method coverage, but does not include a training-budget decomposition. I therefore remain unconvinced that the current evidence sufficiently establishes the broader practical significance of the findings for contemporary image detection research.
I will maintain my rating of 2 (Weak rejection).
Response to Reviewer DNXc: training-budget analysis on AIDE
Thank you for reading our response closely and for the follow-up. Your point about the missing budget analysis on AIDE was a fair one, and addressing it has made the paper better.
We have run the budget analysis the reviewer asks for on AIDE. Further training of the placebo does not close the observed gap: its AUC changes by (seed-level 95% ), and remains.
Setting. We continued the ZERO placebo from its 5-epoch checkpoint, with the same augmentation and validation-loss-based stopping and selection (three seeds; two stopped at epochs 18 and 19, and the third is read at its best validation loss within 18 epochs). The gap also did not shrink at a fixed budget of 18 epochs.
We agree with the reviewer that budget dominance is shown for two systems, and AIDE is not a third. The three detectors give three different readings of the same kind of row:
| Detector | Budget term | Gap to the retrained placebo | Reading |
|---|---|---|---|
| Ours (5 seeds, TPR) | pp of (86%) | budget | |
| LaRE², CVPR 2024 (5 seeds, AUC) | of (71%) | budget | |
| AIDE, ICLR 2025 (3 seeds, AUC) | gap persists |
For AIDE the budget term is the change of the placebo under further training.
Why this matters for contemporary detection research. In all three papers the standard ablation row reads the same way: the evidence branch helps. The analysis shows that the LaRE² row is sensitive to budget and checkpoint rule, whereas the AIDE gap is not closed by further training of the placebo. The controls therefore separate a row that the baseline's budget explains from a gap that survives it, which the row alone cannot do, at a cost of one additional run per seed.
Scope. The paper's claim is that the row is a sum of terms that must be measured separately, not that the budget term dominates everywhere. The abstract's last sentence states this, and the camera-ready subtitle will state the scope as well. We have added the AIDE budget analysis to Sec 5.4 of the revision.
Thank you again for your time and for the careful reading of our code and statistics throughout this discussion. The protocol table, the rule-specific analysis and this AIDE experiment are all in the paper because of your comments.
Response to Reviewer DNXc
Q1. Commit, launch arguments, LR schedule; why validation-best and test-best coincide.
The eight-generator test list was not used. In every training log
--test_fileand--val_fileare the same 2,500-image validation list, so the scheduler stepped on validation AUC. We did not use the released launcher, sotest_scorein the scheduler is validation AUC and no test generator entered the training trajectory.Table I. LaRE² protocol as recorded in the logs (App. F.1).
f8b2966449682845cfbf1bb4ccbb0dd599a686ed--model CLipClassifierWMapV6 --clip_type RN50 --epoches 15 --lr 1e-4 --batch_size 48 --test_batch_size 40 --data_size 224 --val_method con --val_ratio 0 --no_strong_aug--val_file,--test_fileval_2.5k.txtReduceLROnPlateau(mode='max', factor=0.5, patience=4)on validation AUCWe have revised Sec 5.1 to state this configuration.
Q2. Rule-specific seed-level intervals, uncertainty in ρ, coverage.
Done, following the reviewer's split by rule (GR-2). Five seeds: best-validation +0.0147,[−0.0220,+0.0515], ρ(1)=0.083; latest −0.0018,[−0.0155,+0.0120], ρ(1)=0.031. Intervals on ρ(1) are in GR-2, and the seed-level interval covers about 94% in simulation (Table C). Every gap is below ρ(1) of its own rule.
W3. Shared-variance model; ρ as a boundary. We have estimated the variance separately for each rule from five seeds (Table B), and both rules give the same conclusion. ρ is the minimum detectable effect at power 0.8 (L241–242). The sign statement rests on the data: the sign changes with the rule in three of five seeds.
Q3. Does controlling the baseline budget on LaRE² reproduce the principal finding?
Yes. The budget term is 71% of the standard row (+0.0233 of +0.0327 AUC; 70% under best-validation), and the map term is −0.0018,[−0.0155,+0.0120] (Table A).
W1. Contribution beyond established evaluation practice.
Established practice controls the headline score; this paper controls the ablation row, and what it finds there was not known.
What a reader learns that these do not provide:
The reviewer's Strengths name the distinction between using a map and benefiting from it as particularly valuable; items 1–3 are that distinction measured on two published detectors.