- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to recover an observer's similarity metric from sparse triplet judgments. It introduces Observer-Specific Gain (OSG) to measure target agreement relative to agreement with other observers. Its proposed Semantic Factorization and Gated Projection (SFGP) uses self-supervised semantic factorization followed by observer-specific gated projection. Experiments with neural-network observers on CUB, Stanford Cars, and ImageNet-100 show improvements over direct metric learning with limited supervision.
Critical strengths
-
Population-relative evaluation makes a useful distinction between matching a target observer and reproducing judgments shared across observers.
-
SFGP improves target agreement over direct metric learning in several settings, and the component ablations help explain its design.
Critical weaknesses
-
The practical motivation remains unclear. A concrete use case would help explain why an observer-specific similarity metric is needed and what the proposed formulation offers beyond existing personalization or conditional metric learning approaches. Conditional Similarity Networks (Veit et al., 2017) and SCE-Net (Tan et al., 2019) already study different similarity criteria, such as color or category. Although their supervision differs, clearer positioning and comparisons under compatible access assumptions are needed.
-
SFGP is evaluated with only 1,000 or 2,000 training judgments per observer. The gains may primarily reflect better regularization and data efficiency rather than a distinct advantage in recovering observer-specific structure. Data efficiency is valuable, but learning curves extending to 30,000 triplets or more would clarify whether a well-tuned direct triplet baseline catches up.
-
The observers are trained on the same classes with the same objective and differ mainly in sampled images and random seeds. It is unclear whether their disagreement reflects different similarity preferences or ordinary training variation. Testing observers trained on different class sets, or trained to judge similarity by color versus category, would better show whether SFGP captures meaningful differences between observers.
-
Generalization to new images and classes is not clearly established. The paper separates triplets but does not specify whether images overlap across the splits. For the same target observer, training the metric learner on 100 CUB classes and testing on the remaining 100 would test whether its learned metric transfers beyond familiar classes. Test images and classes should be excluded from task-specific training, including factorizer pretraining. Evaluation under domain shifts would also help establish broader generalization.
Questions for authors
-
What concrete application requires an observer-specific similarity metric? How does SFGP differ from CSN?
-
Does SFGP retain its advantage over direct metric learning with substantially more training triplets?
-
Does SFGP capture differences between observers trained on different class sets or trained to judge similarity by different attributes, such as color or category?
-
Are the metric learner's training, validation, and test images disjoint? For a fixed target observer, does the learned metric generalize to held-out classes and domain shifts?