- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies what fixed text anchors contribute to few- and full-shot VLM anomaly detectors with trainable adapters. It theoretically demonstrates that binary predictions depend only on the anchor difference vector. If the adapter is retrained and rotation-closed, the direction of this vector can be injected by an adapter rotation. For a shared multi-category adapter, the invariant extends to the Gram matrix of category-based difference vectors. Extensive experiments show a consistent pattern: once the anchor scale is kept the same, random or orthogonal anchors work about as well as meaningful text prompts. Performance drops when the scale changes, but recovers when the correct scale is restored.
Critical strengths
- The paper asks a simple but important question: are these models actually using the meaning of the prompts, or just their geometry? The controls are carefully designed to separate the two.
- The main theoretical result is clean and easy to follow. It clearly explains why a retrained adapter can absorb changes in anchor direction, while changes in effective scale still matter.
- The experiments are broad and consistent. The same pattern appears across datasets, backbones, training settings, and adapter architectures, which makes the main finding more convincing.
Critical weaknesses
- I could not find enough detail about how the medical datasets are split. In particular, it is important to confirm that images or slices from the same patient do not appear in both the support and test sets.
- Although the paper discusses the limitations of its theory, the title and overall message can still sound broader than the evidence supports. The strongest conclusions apply to fixed binary anchors with a retrained, rotation-closed adapter, rather than vision-language anomaly detection in general.
- In the multi-category experiments, the geometry seed is tied to the training seed. It would be cleaner to vary these separately so that the gain can be attributed more confidently to the anchor geometry.
Questions for authors
- Could you run the multi-category experiments with geometry seeds and training seeds varied independently? This would help separate the effect of anchor geometry from ordinary training randomness.
- Could you clarify exactly how the medical datasets were split? In particular, are the support and test sets fully separated at the patient level?
4 comments
Response to Reviewer MwGj (2/2)
Weakness 3: the geometry seed is tied to the training seed (also Question 1)
"It would be cleaner to vary these separately so that the gain can be attributed more confidently to the anchor geometry"
In short. We varied them separately. The gain holds for every draw of rotations, and the draw matters far less than the gain. Your question also exposed an issue we now disclose: in the original runs, category t used seed s + t, so consecutive geometry seeds shared all but one per-category rotation.
What we ran. Three new geometry seeds whose draws share no per-category rotation, crossed with the same three training seeds (3 × 3 grid). The training seed fixes the support set, initialization and data order; the geometry seed fixes only the rotations. Both banks (shared and independent rotations), MVTec-AD and VisA, all three backbones. A shared-rotation run and an independent-rotation run with the same two seeds form a pair, giving 108 paired comparisons.
What we found (Appendix G.1, Table 40):
- Independent rotations beat the shared rotation in all 108 paired comparisons. Mean gains range from +1.14 to +8.94; the smallest single gain is +1.01, and on DINOv3 VisA all nine pairs gain +8.68 to +9.09 image AUROC.
- The geometry seed moves the means by at most 0.27.
- Its effect is statistically detectable (p < 0.05) only for DINOv2 on VisA, and neither of those two tests survives a correction for the twelve tests.
- The separated means differ from the originally reported ones by at most 0.16.
What this shows. The gain comes from the geometry, not from a lucky draw of rotations: every draw wins, and changing the draw moves the results by less than a tenth of the gain in every setting. The overlap of the original draws did not affect the numbers reported in our paper.
In the revision. §6.4 (one sentence); Appendix A.4 (disclosure); Appendix G.1 (design, Table 40).
Question 1: Could you run the multi-category experiments with geometry seeds and training seeds varied independently? This would help separate the effect of anchor geometry from ordinary training randomness.
Yes, we ran these experiments, and we describe them in our answer to Weakness 3 above. In brief, independent rotations win all 108 paired comparisons, and the geometry seed moves the means by at most 0.27 (Appendix G.1).
Question 2: Could you clarify exactly how the medical datasets were split? In particular, are the support and test sets fully separated at the patient level?
We describe the splits and our audit in our answer to Weakness 1 above. In brief, we use the standard BMAD and MediCLIP benchmark splits unchanged, and the support images for each training seed are drawn from the benchmark's support pool. The splits are not separated by patient for BraTS2021 (abnormal images), LiverCT, RESC and OCT2017, and BUSI, CheXpert and Kaggle BrainMRI carry no patient identifiers. Because the overlap is the same for every anchor condition, it cannot create the equivalence; it does make absolute scores optimistic for all methods, ours and the published ones (Appendix B.2, Table 21).
Follow-up on our responses
Thank you again for your review. As the discussion period ends soon, we would be grateful to know whether our responses address your concerns. If any point remains open, we would be glad to clarify it.
Thanks for the detailed explanation. Most of my concerns have been addressed, and I believe that even though the scope of this paper is limited compared to what the authors claimed, from the theoretical perspective, this is interesting investigation. Therefore, I will keep my current rating.
Response to Reviewer MwGj (1/2)
Thank you for the review. We answer each weakness and each question separately, in your order.
Weakness 1: dataset splits and patient-level separation (also Question 2)
"it is important to confirm that images or slices from the same patient do not appear in both the support and test sets"
In short. We did not make our own splits. We use the standard public benchmark splits exactly as released: BMAD (Benchmarks for Medical Anomaly Detection, Bao et al., CVPR Workshops 2024) for four datasets and the MediCLIP release for the other three. These are the splits on which MVFA, MadCLIP and MediCLIP report their results, so our numbers are directly comparable with theirs. Your question led us to audit these benchmark splits, and the answer is mixed: four of the seven are not fully separated by patient. We now document the splits and the audit in the paper.
How the splits are made. BMAD provides the splits for BraTS2021, LiverCT, RESC and OCT2017, and the MediCLIP release those for BUSI, CheXpert and Kaggle BrainMRI. We change nothing in them. In every run, the training seed draws the K support images at random from the benchmark's support pool, as MVFA does; the test split is used as released.
Audit (Table 21, worst case over all support sets used in our paper):
BUSI, CheXpert and Kaggle BrainMRI carry no patient identifiers. Comparing images directly, we found two exact copies of test images in Kaggle BrainMRI, four near-identical images in BUSI, and none in CheXpert.
What it means for our claim. Every anchor condition uses the same support sets and test images, so the overlap is identical on both sides of every paired comparison and cannot by itself create the equivalence. It could hide a difference only by pushing both sides to the ceiling. It does not on the most affected dataset: LiverCT, split by slice, is the least saturated real-anomaly dataset, and there the scale-only positive control loses 3.9 points while the exact controls stay within ±0.5 (Table 33).
What it means for absolute numbers. Scores on BraTS2021, LiverCT and RESC are optimistic, for our detector and for every published method evaluated on these splits. We did not re-evaluate on patient-disjoint splits; we kept the published ones so that the comparison with published methods stays like for like.
In the revision. §6.1 (one sentence); §7 Limitations; Appendix B.2 (procedure, audit, Table 21); captions of Tables 44–45.
Weakness 2: the title and message sound broader than the evidence
"The strongest conclusions apply to fixed binary anchors with a retrained, rotation-closed adapter"
In short. We agree: the guaranteed conclusions apply to fixed binary anchors with a retrained, rotation-closed adapter, and the message can be read more broadly than that. Beyond that setting we present our results as empirical evidence only.
What the theory covers. Our research question is posed for trained fixed-anchor detectors, and the title is restricted to few- and full-shot detection, where an adapter is trained on support images (not zero-shot). Proposition 1 states where the guarantee stops: adapters that cannot absorb the rotation, frozen adapters, and multi-category replacements that change the joint geometry. The Limitations repeat these requirements.
Inside the theorem, we now test two published detectors, AnomalyGPT and APRIL-GAN, with the same result (see Reviewer sX9b).
Beyond the theorem, the settings we tested show the same pattern. Three lie outside its premise:
Where the claim stops, we say so. With a frozen adapter, as in zero-shot transfer, the anchors it was trained with need to be kept (§6.5).
Changed in the revision.
The title. If the AC and reviewers prefer a narrower title than one restricted to few- and full-shot detection, we will adapt it for the camera-ready version.