acceptodds
Under review as a conference paper at ICLR 2027

Auditing Reference Access in Frozen Feature Anomaly Detection

Abstract

Frozen-feature anomaly detectors are frequently compared unfairly under mismatched reference-access protocols: inductive methods rely on clean normal training images, while transductive methods leverage the full test set that contains anomalies. We present BudgetMatch, an auditable evaluation contract to formalize reference-access specifications and enforce valid single-factor comparisons. With the scoring function fixed, we conduct pre-registered, controlled experiments across MVTec AD, VisA and BTAD to quantify model sensitivity to the budget of clean reference samples. We find that increasing clean reference samples yields consistent gains in macro image-AUROC, and this magnitude of performance shift is comparable to the reported improvements of many published anomaly detection algorithms. The transductive full-pool setting differs across eight protocol dimensions and should be treated as a full-protocol comparator rather than a single-variant baseline. We release machine-verifiable certificates, demonstrating that leaderboard rankings can only be reliably interpreted when reference-access rules are fully disclosed.

Primary area:
datasets and benchmarks
Keywords:
anomaly detection, few‑shot anomaly detection, evaluation methodology, benchmark auditing, frozen features

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iVtyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BudgetMatch, a framework for documenting and checking differences in reference access when comparing anomaly detectors. Controlled experiments measure the effects of reference-bank size, contamination, and source using frozen features. The authors also audit published comparisons and argue that differences in access can be large enough to undermine the interpretation of reported performance margins.

Critical strengths
  1. The paper addresses a meaningful evaluation problem. Methods that use a few verified normal images and methods that access the entire unlabeled test cohort operate under different assumptions. Making these differences explicit helps readers interpret comparisons and judge their relevance to deployment.
  2. The experiments go beyond showing that more references help. The decomposition of reference size, contamination, and source provides useful evidence about why performance changes. Additional scorers, encoders, and datasets strengthen the analysis, and the authors clearly disclose several limitations and negative findings.
Critical weaknesses
  1. The contribution beyond existing evaluation practice is limited. The distinction between inductive and transductive methods is established, and prior work such as AnomalyDINO already separates batched zero-shot evaluation from other settings. The main finding that increasing the number of clean references improves performance is somewhat expected. The proposed cards and validator make reporting more systematic, but checking which declared fields differ is straightforward. The paper needs a clearer explanation of what new understanding this framework provides beyond explicit protocol reporting and standard sensitivity experiments.

  2. The headline comparison depends heavily on the scoring rule. With nearest-neighbor scoring, one clean reference matches or beats the full test cohort in 15 of 27 categories. Against official MuSc, this drops to only 4 categories, and even 16 clean references are insufficient in 8. This matters because nearest-neighbor scoring is particularly vulnerable to anomalous references: a single similar defect can suppress the anomaly score.

Questions for authors
  1. What is the main new insight beyond the known benefit of more clean references and the distinction between inductive and transductive evaluation?

  2. Have you tested whether giving existing methods the same reference data changes which method performs best?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iVty

Thank you for the review and for Q2, which we have now answered with new runs.

W1/Q1. "What is the main new insight beyond the known benefit of more clean references and the distinction between inductive and transductive evaluation?"

The new insight is that the access fields a setting label leaves undeclared can be found and priced, and that setting this price against a printed margin shows whether a published ranking survives that field. The inductive/transductive distinction itself is established: the submission cites Bronskill et al. (2020), and the revision adds, in Section 2, their point that transductive methods "must be labeled as such". As you note, AnomalyDINO separates batched zero-shot evaluation; it places MuSc and ACR there, which removes the label ambiguity for them. Its few-shot Table 3, however, still places APRIL-GAN, which trains on another dataset's labeled test set and normalizes over the test set, beside training-free methods such as WinCLIP+ and its own AnomalyDINO-S. What we add is:

  • A field-level check whose questions surface access differences that a setting label, and sometimes the method's own paper, do not show. APRIL-GAN's test-set normalization is one.
  • A price, in clean reference images, that changes with the scorer. Under a nearest-neighbor (1-NN) scorer the test cohort is worth at most one clean image in 15 of 27 categories, under official MuSc in only 4.
  • A test of what an undeclared field does to a published ranking. Across 30 audited comparisons, none is undecidable, 12 are uncertain and 18 decidable (General response 1/2; Appendix Z).

W2. "This matters because nearest-neighbor scoring is particularly vulnerable to anomalous references: a single similar defect can suppress the anomaly score."

We agree. In the source × purity decomposition the cohort's anomalies cost +3.61 and +7.23 points (MVTec AD and VisA), against +0.41 and +0.50 for its source. Measured on a cohort subsample, the purity cost is +3.58 and +7.45 under 1-NN and falls to +2.35 and +4.19 under MuSc's interval average. In the 8 categories where even 16 clean images under our scorer do not reach official MuSc on the test cohort, the comparison changes the scorer as well as the access (Appendix O), so we read them as scorer dependence, not as a single-factor price. The revised Section 5 gives steps for pricing a field in each detector's own pipeline (response to Reviewer erdj, 2/2).

Q2. "Have you tested whether giving existing methods the same reference data changes which method performs best?"

Yes. With access matched on clean reference images, none of the columns we re-ranked in three published tables has a single identifiable leader. Aligned APRIL-GAN runs without its auxiliary-trained branch and test-set normalization. Two rows cannot be told apart when the 95% interval of their difference includes 0 or, against a printed row, when the difference is within the larger of our range across support paths and the printed standard deviation.

Published tableMatched comparisonDifference (95% CI where shown)Reading
MuSc Table 1, four shots, MVTec ADaligned APRIL-GAN vs the patch-level functional MuSc uses (our 1-NN, four clean images)+0.72 [−0.95, +3.17]cannot be told apart
MuSc Table 1, four shots, VisAsame+1.19 [−0.79, +3.52]cannot be told apart
AnomalyDINO Table 3, VisA, 1/2/4 shotsaligned APRIL-GAN vs AnomalyDINO-S (672-pixel input)+1.21, +0.89, −0.35cannot be told apart at any of the three
InCTRL Table 1, MVTec AD, 2/4/8 shotsInCTRL without its trained branch vs WinCLIP as printed+0.99, +1.12, +0.46cannot be told apart at any of the three

For MuSc's Table 1, we restricted the compared methods to four clean training images; MuSc itself cannot run under this restriction, since its scoring needs the test set. Aligned APRIL-GAN cannot be told apart from MuSc's printed row on either benchmark, nor, on MVTec AD, from WinCLIP; on VisA it stays above PatchCore and WinCLIP. Neither column has a single leader.

The other reading of "the same reference data" gives every method the whole test set. Official MuSc against our 1-NN on the same full test set (self-excluded) differs by +1.71 [−1.23, +4.53] on MVTec AD (cannot be told apart) and +4.54 [+1.87, +7.32] on VisA (MuSc leads). The new Appendix Z gives the orderings, the released R1_aprilgan_analysis.json and R1_inctrl_analysis.json give the numbers, and the revised Appendix U gives the whole-test-set comparison.

We would be glad to answer any further questions.

Reviewer_iVtyOpenReview ↗

Response to the authors

Thanks for providing the extra evidence. Seeing how much reference access mattered in the published comparisons helped me understand what the audit adds and the revision addresses my main concern. Based on that I’m raising my score

Thank you for reading our response and for raising your score. The evidence on how much reference access mattered in the published comparisons is part of the revision, and it will stay in the final version.

Reviewer_erdjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BudgetMatch, a framework for auditing reference access in industrial anomaly detection. It uses protocol cards and comparison certificates to identify differences in evaluation settings. Experiments show that reference access substantially affects detection performance and that existing benchmarks often compare methods under different protocols.

Critical strengths
  1. The paper addresses an important issue in anomaly detection benchmarking, highlighting how differences in reference access can undermine fair comparisons between methods.

  2. The experimental analysis is carefully controlled and provides convincing evidence that reference budget and purity can substantially affect anomaly detection performance.

  3. BudgetMatch provides a structured and auditable framework for identifying differences in evaluation protocols, which could improve the transparency and reproducibility of future comparisons.

Critical weaknesses
  1. The novelty over existing evaluation frameworks is limited. Evaluation Cards (Ghosh et al., 2026) and Siravuri & Alhoori (2026) already record evaluation differences and group results by protocol. The paper does not show what the field-level check and machine-readable certificate add in practice.

  2. The value of the full test cohort depends heavily on the scorer. Under 1-NN, one clean image matches or beats the cohort in 15 of 27 categories, but only in 4 under official MuSc. Under MuSc's interval average, the advantage of a same-size clean bank on MVTec AD also becomes undetermined. So it is unclear how far the findings carry over to other detectors.

  3. The support for C2 is not fully convincing. The paper compares the 1-to-16 clean-image gain with gains reported in prior papers. But shot counts are usually stated in published tables, while the paper's concern is undeclared access differences. The paper does not directly establish whether these undeclared differences in the audited comparisons have effects comparable to the reported method gains.

Questions for authors
  1. Can the authors provide a concrete example where BudgetMatch identifies a comparison issue that existing evaluation frameworks would miss?
  2. Given the strong dependence on scoring functions, how should researchers use the measured reference-access effects when comparing different detectors?
  3. Can the authors examine whether the protocol differences identified in the audit actually affect the reported method rankings, particularly when additional training or labeled auxiliary data are involved?
2 comments
AuthorsOpenReview ↗

Response to Reviewer erdj (1/2)

Thank you for the review and for the three concrete questions.

W1/Q1. "Can the authors provide a concrete example where BudgetMatch identifies a comparison issue that existing evaluation frameworks would miss?"

Yes. APRIL-GAN's released code min-max normalizes each few-shot image score over the category's whole test set, so one image's score depends on the other test images. Its paper does not state this, and neither of the two frameworks you name, nor the setting labels of the tables that print APRIL-GAN, record it. Our cards ask whether a score depends on other test queries; answering that from the released code is how we found it. Coding it raises the comparisons with a test-time access difference in Appendix G from 6 to 24 of 54.

BudgetMatch also differs from the two frameworks in what it compares and when:

  • Evaluation Cards flag only reports of the same model on one benchmark and metric whose scores diverge by more than 5% of the metric range. Their operational fields contain no reference or training-data access, query visibility, self-exclusion or batch coupling (splits and contamination are items to report, not computed signals), and they target LLM evaluation.
  • Siravuri & Alhoori group results by an observed protocol signature (split, metric name and unit). Their own validation finds only 2 of 15 same-signature pairs truly comparable, noting that same-signature entries "are still frequently incomparable on hidden facets". In their released data, the MVTec AD PRO leaderboard places MuSc (93.8) in one comparable cluster with the 72.7 that MuSc's abstract compares against, which MuSc's Table 1 shows to be ACR. MuSc scores against the rest of the test set and ACR against a test batch, and nothing flags this.
  • BudgetMatch compares the access fields of two different methods before any score is read. The revised Section 2 now states that neither framework compares reference access, query visibility or batch coupling.

What the certificate adds in practice. The validator flattens the four cards of two protocols, compares every field, and requires the set of fields that actually differ to equal the set the certificate states and to lie inside the declared intervention set; one field too many or too few against the stated set is an error. This is how the submission certifies the budget study (RQ1) as differing in exactly one field, and the comparison with the full-pool protocol (RQ2) in eight. For the literature audit, the released recompute_multitable.py derives the difference set, verified difference and reversal status of all 66 comparisons from the coded cards, so a reader can change one coded value and see which counts move. For MuSc's Table 1 (MVTec AD, "0-shot"; B01 in the released coding file), the script's pair-level output (reformatted) shows that MuSc differs from WinCLIP in eight fields and from ACR in six, although all three share the zero-shot label:

MuSc vs WinCLIP: verified difference = access.source; access.purity; access.query_visibility;
  access.self_exclusion; access.cohort_coupling; resource.reference_budget;
  resource.selection; resource.reuse_update
MuSc vs ACR:     verified difference = access.source; access.purity; access.self_exclusion;
  resource.reference_budget; resource.selection; resource.reuse_update

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer erdj (2/2)

W3/Q3. "Can the authors examine whether the protocol differences identified in the audit actually affect the reported method rankings, particularly when additional training or labeled auxiliary data are involved?"

Yes. We measured the undeclared fields themselves (General response 1/2).

In 15 of the 30 measured comparisons, the field separating a single leader from its comparator is training on labeled auxiliary data, through a component that enters the score (APRIL-GAN, InCTRL, AnomalyCLIP's learned prompts). Of these, 7 rankings are decidable and 8 uncertain. Three cases show the range:

  • APRIL-GAN's VisA lead over WinCLIP+ survives removing its auxiliary-trained branch (decidable in all five comparisons: B18–B20 in APRIL-GAN's table and B28–B29 in AnomalyDINO's).
  • On MVTec AD two-shot, removing the same branch raises APRIL-GAN by more than WinCLIP's 2.0-point lead, but the interval reaches below that lead (uncertain).
  • InCTRL's trained branch is worth about zero on MVTec AD, where the margins are trivial.

Matching access can also leave a column without a single identifiable leader. Without its auxiliary-trained branch and test-set normalization, APRIL-GAN can no longer be told apart from AnomalyDINO-S (672-pixel input) on VisA at one and two shots, where it led as printed.

W2/Q2. "Given the strong dependence on scoring functions, how should researchers use the measured reference-access effects when comparing different detectors?"

By measuring the field in each detector's own pipeline. The dependence on the scorer is part of our finding (C1: what a reference-access difference is worth, priced in clean reference images, depends on the scorer). The revised Section 5 sets out these steps, and Appendix Z applies them:

  1. If the differing field exists on one side only, measure its worth in that side's own pipeline (or the worth of the component that exists only because of it).
  2. If both sides have the field with different values, measure both and set the smaller worth against the margin.
  3. If a side cannot be measured, say so; a conclusion then holds only for the side that was measured.
  4. Read the result with the three labels: undecidable if the worth's 95% interval lies at or above the margin, decidable if below, uncertain otherwise.

To show how much the effect moves, the revised Appendix U (new Table 40) gives the one-to-sixteen clean-budget effect under each detector's own pipeline. Three of its rows, in image AUROC points:

DetectorMVTec ADVisA
Our 1-NN (OpenCLIP ViT-L/14-336)+5.60+6.31
AnomalyDINO (DINOv2 ViT-S/14, 448-pixel input, no masking)+1.74+8.93
MuSc+ (as printed)+5.3—

Across these three detectors the effect ranges from about 2 to 9 points, and the same detector can differ greatly between datasets. So when two detectors are compared, the price has to be measured in each one's own pipeline. Under MuSc's interval average, the clean bank's advantage on MVTec AD becomes undetermined, as you note; this is the same scorer dependence.

We would welcome further questions on either point.

Reviewer_9BDhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BudgetMatch describes anomaly-detection protocols using four cards: access, resource, functional, and evaluation. A validator lists the fields that differ between two protocols. The main experiment fixes a frozen encoder and a 1-NN scorer. It varies clean reference budgets over 1, 2, 4, 8, and 16 images on five nested support paths. The 702-cell study reports gains of 5.60 and 6.31 AUROC points from one to sixteen images on MVTec AD and VisA. Further experiments test other scorers, encoders, and datasets. A literature audit finds different access under one setting label in 54 of 66 comparisons.

Critical strengths
  1. The protocol specification makes comparison assumptions explicit. It separates clean support access from access to other test queries. The difference-set rule also separates a single-factor intervention from a comparison of complete protocols (Sec. 3.1). The coded examples show why a shared zero-shot label is insufficient. This gives authors and reviewers a concrete reporting check.

  2. The budget result has useful controls and extensions. The primary study fixes the scorer and uses nested supports. All ten dataset-path endpoint contrasts are positive (Table 2). The subspace scorer also gives ten positive contrasts (Table 34), and Real-IAD shows the effect without saturated categories (Tables 35–36). The MuSc reduction tests show where the clean-bank advantage becomes smaller. These results support a broader finding than a single 1-NN experiment.

Critical weaknesses
  1. The C2 ranking rule is not validated on a matched published comparison. Table 38 compares the one-to-sixteen clean-budget effect with 46 proposal gains across different scorers and protocols. Appendix G states that 48 of the 54 comparisons with verified differences have source or purity differences but no test-time access difference. Eleven of the 12 robust “reversals” concern extra labeled training data. The measured clean-budget effect does not quantify those differences. The audit supports better disclosure, but it does not show that the measured price determines any specific audited ranking. This limits the main practical claim in Sec. 5.

  2. The described release leaves key evidence unavailable for independent verfication. Appendix X says the registration documents are withheld from the anonymous artifact. The commitments have no third-party timestamps. Appendix W says the raw-score archive is identified by hash rather than redistributed. The artifact can regenerate tables from accepted aggregates, but repeating the reported raw-to-metric check requires that archive. The reported checks are useful. Still, reviewers cannot inspect the original registered decisions or repeat that check from the described release alone. Auditability is a central contribution, so these gaps matter.

Questions for authors
  1. Can you show one audited comparision where the actual differing field is measured under the relevant scorer and protocol? Please compare that effect with the printed margin and show how it supports the C2 verdict.

  2. Can you provide anonymized registration documents, their available timing records, and the raw-score inputs needed for the metric audit? This would allow checking the evidence described in Appendices W and X.c

  3. Was the coding of the 374 rows independently checked by a second coder? Please report field-level agreement and how disagreements were resolved. The sensitivity codings in Appendix G are useful, but coder agreement is not reported.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9BDh (1/2)

Thank you for the precise reading of original Table 38 (now 39) and Appendix G. We agree with W1, and Q1 asks for exactly the test we have now run.

W1/Q1. "Can you show one audited comparison where the actual differing field is measured under the relevant scorer and protocol? Please compare that effect with the printed margin and show how it supports the C2 verdict."

Yes. We did this for 30 audited comparisons (General response 1/2; new Table 4, Appendix Z and Table 45). Three bear directly on your points:

  • Robust reversals from extra labeled training (APRIL-GAN vs WinCLIP+, VisA; B18–B20). We reran APRIL-GAN's released code on our five nested support paths, with and without the linear branch trained on another dataset's labeled test set. The branch is worth +0.3 to +3.2 points across the pre-specified versions, against margins of 7.4, 7.6 and 5.3. The 95% intervals lie below the margins, so the rankings are decidable and the branch does not explain the lead.
  • The same field, against a smaller margin (WinCLIP vs APRIL-GAN, MVTec AD two-shot; B15). Removing the same branch raises APRIL-GAN by 2.8 to 3.5 points, more than WinCLIP's 2.0-point lead. Every interval still reaches below the margin, so the ranking is uncertain.
  • Extra labeled training in InCTRL (MVTec AD; B44, B46, B48). InCTRL's trained branch is worth −0.3 to +0.3 points. The intervals straddle margins of 0.9, 0.5 and 0.6, which are themselves within the printed noise, so these rankings are uncertain.

Across all 30, 12 rankings are uncertain (7 with non-trivial margins), 18 are decidable and none is undecidable. Of the 12 robust reversals you cite, 6 are decidable and 3 uncertain with trivial margins. The other 3 (InCTRL on VisA) cannot be run, because the checkpoints trained on MVTec AD are no longer distributed; their margins are also within the printed noise. We have narrowed C2 and the practical claim in Section 5. The revised practical claim is that an undeclared field should be measured in the pipeline of the method that has it and compared with the margin, and that until then the row does not support a claim about the method alone.

W1 (cont.). "Appendix G states that 48 of the 54 comparisons with verified differences have source or purity differences but no test-time access difference."

We agree that the clean-budget effect does not price these differences, which is why the new test measures the differing field itself (above). The count of 48 is coded from the papers. APRIL-GAN's released code normalizes each few-shot image score over the whole test set, which its paper does not state. With APRIL-GAN coded from its released code, 30 of the 54 have source or purity differences but no test-time access difference and 24 have a test-time access difference, and 6 reversals then also involve a test-time field. Appendix G now gives both codings.

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer 9BDh (2/2)

W2/Q2. "Can you provide anonymized registration documents, their available timing records, and the raw-score inputs needed for the metric audit?"

We release the raw scores, the timing records and the three archived runners whose hashes the registration ledger prints in a new folder rebuttal/ of the supplementary material, with a README that describes each part. The registration documents are internal records, identified by their hash prefixes in original Table 40 (now 42). We did not attach the supplementary material at submission; the artifact the paper describes is now attached alongside rebuttal/.

  • Raw scores and the metric check. For all 702 cells we release image-level scores, labels and query paths, the metrics.json files, and a script that recomputes image AUROC, AP and F1-max per cell and compares them with those files. The script also checks the hashes of each cell's metrics.json, identity.json and config_card.json against the manifest of the supplementary material. Run in a clean directory on the released files alone, it matches all 702 cells with zero difference and all 2106 hashes.
  • Runners and protocol R1. The three runners match their printed prefixes (verify_prefixes.py checks this), and protocol R1 matches the hash we posted in this forum.
  • Timing records. Our server inventories and the append-only run ledgers of the main 702-cell run and of four decomposition runs. For the new R1 experiments, we posted the protocol's hash in this forum on Nov 10, before any of its runs.

For the new C2 experiments, R1's support and calibration manifests, code patches, per-image scores and analysis scripts are also released, and the scripts reproduce the analysis results from the released scores. The revised Reproducibility Statement and the README of rebuttal/ describe this release.

Q3. "Was the coding of the 374 rows independently checked by a second coder? Please report field-level agreement and how disagreements were resolved."

No. One author coded the 374 rows, so there is no inter-coder agreement to report. The released coding can be checked and recomputed:

  • Every value has a locator. The coding file in the supplementary material gives an evidence locator for every coded field, and recompute_multitable.py recomputes from that file the comparison, pair and reversal counts of Appendix G under the primary and the three sensitivity codings. A reader can check any value and see which counts it moves.
  • Sensitivity codings. Appendix G gives the counts under three alternative codings: printed attribution kept for values traced to another shot count, anomaly-type names ignored, and the two known support-draw rules treated as one. The 12 robust reversals are those that hold under all four codings.
  • A coding from released code. The revised Appendix G adds a coding of APRIL-GAN from its released code (W1, in part 1/2). It moves the split of test-time differences but no verified, reversal or robust count.

We are glad to provide anything further you need to check.

Thanks for measuring the undeclared fields on 30 audited comparisons and releasing the raw scores, this answers my main concern on C2 so I raise my score to 3.

Thank you for reading our response and for raising your score. The measurements of the undeclared fields on the 30 audited comparisons and the raw scores are part of the revision, and they will stay in the final version.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): testing C2 on audited comparisons

We thank the three reviewers for careful and constructive reviews. Reviewers found that the paper "addresses an important issue in anomaly detection benchmarking" (erdj), that "making these differences explicit helps readers interpret comparisons" (iVty), that the protocol specification "gives authors and reviewers a concrete reporting check" (9BDh), and that the budget experiments are "carefully controlled" (erdj) with "useful controls and extensions" (9BDh).

The main shared concern is C2, our claim that the worth of an undeclared access field is as large as most margins that published tables rank by. Reviewers 9BDh and erdj note that original Table 38 (now 39) sets the one-to-sixteen clean-budget effect beside gains printed under other scorers and protocols, and does not measure an undeclared field inside an audited comparison. We agree, and we have now tested C2 directly. Reviewer iVty asks a related question, whether matching the reference data changes which method is best; we answer it in our response to Reviewer iVty.

What we measured. For 30 of the 66 comparisons audited in Appendix G, we measured the worth of the field that separates the leader from its comparator. A field's worth is the AUROC difference, in points, between running the method that has the field with it and without it, signed so that a positive worth favors the leader; we measured it mostly by rerunning that method. We then compared the category-bootstrap 95% interval of this worth with the printed margin:

  • undecidable: the interval lies at or above the margin, so the field alone could account for the lead;
  • decidable: the interval lies below the margin, so the field does not account for the lead;
  • uncertain: the interval straddles the margin.

Where a field can be removed in several ways, all versions were written down before the label was computed, and we keep the label least favorable to C2. Eight of the 30 follow protocol R1, whose hash we posted in this forum on Nov 10, before any of its runs.

What we found. IDs such as B15 refer to the coding file in the supplementary material; the new Table 45 maps each measured comparison to its paper and table.

LabelComparisonsExample
Undecidable0—
Uncertain12 (7 with non-trivial margins)WinCLIP vs APRIL-GAN, MVTec AD two-shot (B15)
Decidable18APRIL-GAN vs WinCLIP+, VisA (B18–B20)

A non-trivial margin is larger than both rows' printed standard deviations and 1.0 point; in those 7 comparisons the undeclared field may decide the ranking. In the decidable example, APRIL-GAN's VisA leads of 7.4, 7.6 and 5.3 points survive removing its branch trained on labeled auxiliary data (worth +0.3 to +3.2), and APRIL-GAN still leads by 4.8 to 6.0 points once that branch and its normalization of each score over the test set are both removed.

We have narrowed C2 accordingly. The claim is now that a setting label cannot stand in for the worth of the fields it leaves undeclared, and that whether a ranking survives that worth has to be measured comparison by comparison. The new main-text Table 4 gives the summary. Appendix Z, Table 45 and the supplementary file c2_all_rows.csv give every comparison and every pre-specified version.

(continued in 2/2)

AuthorsOpenReview ↗

General response (2/2): changes in the revision

We have uploaded the revised PDF and supplementary material. The main changes, by location:

  1. C2 tested and narrowed (all reviewers). Abstract; C2 in Section 1; the C2 paragraph of Section 4.2 with the new Table 4; Sections 5 and 6; the new Appendix Z with Table 45; Table 44 in Appendix X. Original Table 38 (now 39) stays in Appendix U, captioned as descriptive.
  2. A test-time dependency found in released code (Reviewer 9BDh, W1; Reviewer erdj, Q1; Reviewer iVty, Q1). APRIL-GAN normalizes each image score over the test set. Coding this from the code rather than the paper raises the comparisons with a test-time access difference from 6 to 24 of the 54 in Appendix G, which now gives both codings.
  3. Sharper comparison with Evaluation Cards and Siravuri & Alhoori in Section 2 (Reviewer erdj, W1/Q1).
  4. How to use a scorer-dependent price (Reviewer erdj, W2/Q2; Reviewer iVty, W2). Section 5 now says in which detectors to measure a field's worth and what to report when one side cannot be measured. Appendix U adds the one-to-sixteen budget effect under each detector's own pipeline (new Table 40).
  5. How the coding was done and can be checked (Reviewer 9BDh, Q3). Appendix G now states who coded the rows and how its comparison, pair and reversal counts can be recomputed from the coding file in the supplementary material.
  6. Evidence release (Reviewer 9BDh, W2/Q2). A new folder rebuttal/ in the supplementary material holds image-level scores for all 702 cells with a script that recomputes every image metric; the three archived runners whose hashes Table 42 prints; our server inventories and run ledgers; and, for the new C2 experiments, protocol R1, code patches, per-image scores and analysis scripts.

Details are in the responses to each reviewer. Table numbers shift by one from original Table 4 and by two from original Table 39 on (original Tables 12, 34, 35–36, 38 and 40 are now 13, 35, 36–37, 39 and 42). Appendix letters are unchanged, and Appendix Z is new.

Please let us know during the discussion if any of these changes needs clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.