- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper trains a small head for each entity on top of a frozen image encoder. Each entity uses 10 target images and 10 non-target images, with a two-group contrastive loss. The head is then used to rescore the gallery. The same heads are evaluated on two things. The first is retrieval (Recall@K over 100 entities). The second is 304 human-annotated image pairs (AUROC for semantic similarity and visual similarity). Across all six combinations of CLIP, DINOv3 and SigLIP with linear and MLP heads, Recall goes up and semantic AUROC goes down slightly. The paper proposes a per-entity scalar that mixes the frozen score and the adapted score, and uses it to choose an operating point. The authors say this works on CLIP and does not work on DINOv3.
Critical strengths
- Evaluating the same heads on two dimensions is a reasonable design, and the main numbers check out. Tables 1-4 match the results file shipped with the supplementary material. The manifests show no image overlap between the image-pair set and the retrieval images. The code does what Sec. 4 describes.
- The authors are fairly candid about the limitations. Only a single seed is used, Table 8 gives per-entity results, the margins are pre-registered, and the effect sizes are not overstated.
Critical weaknesses
-
Writing and figures. The paper is hard to read. The setup, definitions and key concepts are scattered across different sections and appendices, and the logic does not flow. Some sentences look like internal notes that were not fully removed ("taken from the shipped checkpoints rather than from the design document", p12, and "accompany Appendix G and Appendix G", p17). PCSF is never spelled out, and the epsilon in Eq. 8 is never given. Figure 1 and Figure 2 are schematics built from icons. Neither of them clearly conveys the motivation or the method. The paper does not contain a single example of a real retrieval result. For a paper about operator-defined entities, this is strange. I had to read the code to understand what they actually did.
-
Method. The way the method is built already determines that it will show the trade-off the paper reports, and the paper offers no way to deal with it. The two-group contrastive loss pulls ten unrelated non-target images into the same cluster. This is bound to distort the geometry of the similarity space. The authors say so themselves in Sec. 4.2, but they keep the loss anyway. The head projects 768 dimensions down to 128, so information is lost before training even starts. The only "control" mechanism is a per-entity linear interpolation between the frozen score and the adapted score. The authors acknowledge that this is structurally WiSE-FT (Sec. 2), and later say it is not their proposed method (Sec. 6.3, Sec. 8). So the paper designs a head that is bound to lose generality, measures that it loses generality, and in the end does not give the reader a method. The finding itself (specialising a representation costs generality) is already known from the feature-distortion literature that the paper cites.
-
Experiments. The controls needed to attribute the effect to the adaptation are all missing. There is no untrained head, no random 768-to-128 projection, no label-shuffled head, and no prototype baseline on the same 10+10 images. The authors themselves say in Sec. 2 that a prototype is the most direct comparison. All heads start from the same seed, and the whole paper uses only one seed. The effect is also small. Semantic AUROC drops by 0.005 to 0.016, which falls within the authors' own pre-registered margin of 0.010 (Table 14). In addition, the results on "controllability" contradict each other. Fig. 3b shows R@5 of about 81.8 at alpha=1, while Table 1 reports 86.75. The alpha selected by outer-CV is 0.05, and the held-out gain is zero (Table 11).
Questions for authors
- Does the AUROC drop still hold up after a pair-level bootstrap or a DeLong test?
- Do the fused rows and the alpha=1 rows in Table 9 come from the same set of heads? Why does Fig. 3b give about 81.8 at alpha=1?
- How are the negatives chosen? What happens if the negative pool shares no identities with the gallery?
- Please show real retrieval results. Pick a few entities and show, side by side, the query, the top-k images returned by the frozen encoder, and the top-k images returned by the adapted head. Mark the non-target images that are easy to confuse. Right now there is no way to see what the head actually changes.
3 comments
I thank the authors for a detailed and honest response. The revision resolves two of my concerns. The paper is now readable, and most of the controls I asked for were run and reported as they came out: same-support baselines (Table 1), five-seed matched controls with untrained heads (Table 3), and intervals that account for pair dependence. Several concerns remain. The training hyperparameters of the canonical heads differ between the original paper and code (AdamW, learning rate 5e-5) and the revision (Adam, learning rate 1e-4), while the reported Recall values are unchanged. Negative supports share identities with gallery distractors in all 300 units, and no identity-disjoint control was run. The response says the source of the earlier ~81.8 endpoint is still unidentified. More importantly, the new controls did not confirm the original claim. They replaced it. A positive prototype and logistic regression on frozen features both beat the 128-D heads, and on point estimates most of the similarity change is already present in the untrained projection. The submission I reviewed claimed a controllable trade-off of a per-entity head, and the revision claims that standard frozen-feature baselines beat that head, with a new title and 35 pages in place of 20. I think this goes beyond what a revision in the discussion period should do, and the new paper would be better served by a fresh submission with a full review. I raise my rating from 1 to 2.
Response to Reviewer hi1r – Follow-up
Thank you for your thoughtful follow-up and for raising your rating. We appreciate your recognition of the improvements in readability and experimental controls. We also acknowledge that your remaining concerns are substantive.
1. Optimizer discrepancy and provenance
We investigated the discrepancy between the original submission, the distributed supplementary code, and the revised manuscript. The original submission and supplementary training recipe specified AdamW (learning rate , weight decay , gradient clipping 0.85). However, archived trainer-initialization logs from June 2026 record Adam (learning rate , weight decay ). The earlier trainer implementation and a September 2026 SigLIP experiment report also support Adam with these settings and gradient clipping at 1.0. Locally preserved manuscript source versions likewise specify Adam.
We acknowledge that the original submission and supplementary materials contained inconsistent training descriptions. We have corrected the supplementary training code, configurations, and documentation to reflect the settings supported by the execution records.
The reported canonical Recall values are unchanged because we did not retrain the canonical heads for this correction; we retained the previously reported results. Importantly, the available records do not link every individual checkpoint to its optimizer configuration. We therefore regard Adam as the best-supported configuration for the recorded training runs, but do not claim complete checkpoint-by-checkpoint provenance verification. We regret the original inconsistency.
2. Remaining experimental limitations
We agree that the identity-disjoint negative-support control remains missing. Our retrospective audit found filename-derived identity-proxy overlap with the corresponding galleries in all 300 target-repeat units, but these proxies were not independently verified as semantic identities. We found no exact support/gallery image-ID or hash matches and no known identity-disjoint replacement candidates in the evaluated pool. We do not claim generalization to unseen negative identities, and this limitation remains explicit in the revision.
The source of the earlier approximately 81.8% interpolation endpoint also remains unidentified. Revised Figure 4(b) was regenerated from the canonical raw-score series, and its CLIP linear endpoint at now agrees with the canonical Recall@5 of 86.75%. This corrects the current figure's consistency but does not explain the earlier discrepancy. We do not present its origin as resolved.
3. Interpretation of the revised contribution
We agree with your central observation: the additional controls did not confirm the original interpretation; they required us to change it.
The 128-D heads improve retrieval over frozen cosine ranking, but the new same-support comparisons show that logistic regression on frozen features outperforms the trained MLP on all three encoders. The matched controls further indicate, on point estimates, that much of the narrow-head semantic-AUROC gap is present before training.
Accordingly, we no longer claim that the per-entity head is superior to standard same-support baselines or that interpolation establishes a controllable trade-off. We have reframed the contribution as a diagnostic comparison, with narrower conclusions and explicit limitations.
We acknowledge that this is a substantial change in the paper's interpretation and scope. We understand your concern that such a revision may warrant a fresh submission and full review. We respect the Area Chair's judgment regarding how the revision should be treated within the current review process. We do not ask that the new controls be regarded as confirmation of the original claim.
Thank you again for your constructive criticism and for acknowledging the improvements.
Response to Reviewer hi1r
We thank the reviewer for the detailed critique. We agree that the original paper was hard to follow and lacked controls separating projection from training. We restructured the Introduction, protocol, results and limitations, clarified the role of PCSF, and removed internal-note text.
1. Presentation and retrieval examples. Figure 1 now shows top-5 rankings for three real queries as T/L/O/U-labelled outcome tiles (target/look-alike/other entity/unrelated), comparing frozen, MLP and logistic regression. Appendix I.1 (Table 22) describes each non-target item. We cannot redistribute the third-party images or entity names, so this is evidence from real rankings, but it does not fulfill the request for side-by-side image examples.
2. Same-support and untrained controls. We added positive prototype, positive-negative prototype, query expansion and logistic regression on frozen features using the same enrollment support set (Section 6.2, Table 1; Appendix I). Logistic regression exceeds the trained MLP in Recall@5 by +6.17 [95% CI +4.17,+8.33], +10.08 [+6.00,+14.17], and +3.92 [+2.00,+6.00] percentage points on CLIP, DINOv3 and SigLIP. This concerns registered-entity membership retrieval, not query-conditioned visual search.
Five-seed matched controls evaluate initial and trained heads, identity/random 768-D heads, random/transductive-SVD 128-D projections and 128-768-D widths (Section 6.4; Table 3; Appendices H/J). On CLIP, the trained identity-initialised 768-D head gains +6.70 Recall@5 points with a -0.04-point semantic-AUROC change; the random 128-D linear head already has a -0.74-point semantic-AUROC difference before training (95% CI [-1.66,+0.10]). Point estimates suggest much of the narrow-head gap precedes training, although most initial intervals include zero. The matched protocol uses cached features and 30 updates, unlike canonical training. SVD is transductive. We did not run a label-shuffled-head control.
3. Statistical uncertainty and loss. We excluded two support-content-overlap pairs, leaving 302 pairs and 187 image-connected components. Bootstrap intervals resample targets and pair components, and also training seeds for matched controls (Sections 5-6.4; Appendices G/H/J). Canonical heads remain single-seed. On CLIP, semantic-AUROC changes are -0.81 [95% CI -1.25,-0.42] for Linear and -1.44 [-3.26,+0.37] points for MLP. We do not report a DeLong test. The +/-1-point descriptive margin was set post hoc: inclusion within it does not establish zero effect. Some matched training-effect intervals exclude zero.
In loss ablations, SupCon-pos improves Recall@5 over SupCon in all nine tested encoder/head conditions; Pair-BCE is lower in seven (Appendix J, Table 23). SupCon-pos also removes negative-anchor terms, so it does not isolate negative-negative attraction as a single cause.
4. Interpolation and prior inconsistency. We now separate descriptive full-pair sweeps from image-component-disjoint CV and target-level outer CV (Appendix K). Revised Figure 4(b) uses canonical raw scores: at alpha=1, CLIP Recall@5 is 86.75% (Linear) and 87.33% (MLP), agreeing with the main tables. The source of the old Figure 3(b) ~81.8% endpoint remains unidentified. The deployed multi-feature system is a distinct pipeline, not evidence for the canonical comparison. Interpolation/PCSF is exploratory: outer CV chooses alpha=0 in 138/150 cells, with a +1.37-point gain for CLIP MLP but no established robust operating point.
5. Negative identities and remaining limits. A retrospective filename-derived proxy audit finds proxy-identity overlap with the corresponding galleries in all 300 target-repeat units (3,000 negative-support uses, 300 unique image IDs, 100 proxy identities, and 12,000 different-image proxy matches), but no exact image-ID/hash overlap (Appendix J, Table 28). Proxies are not human-verified; no known disjoint replacement was available in the evaluated pool, and 1,500 gallery images lacked usable identity assignments. We therefore did not run identity-disjoint retraining. Generalization to unseen negative identities and catalog-scale effects remain untested.
The revised contribution is a diagnostic comparison under this protocol, not a new adaptation method or a universal trade-off claim.