Observer-Specific Metric Learning via Semantic Factorization and Gated Projection

Similarity judgments can vary across observers: observers may emphasize different attributes of the same data and thereby induce distinct similarity metrics. Yet conventional metric learning does not explicitly distinguish observer-specific structure from similarity shared across a population. We address two aspects of this problem. First, we formulate observer-specific metric learning (OSML) as recovering a target observer's similarity metric from sparse relational judgments. To distinguish observer-specific learning from population-shared predictability, we introduce a population-aware evaluation framework and Observer-Specific Gain (OSG), which compares agreement with the target observer against agreement with non-target observers. Across datasets and neural-network observer populations, population diversity relates to cross-observer predictability, supporting the need for population-aware evaluation. Second, we propose Semantic Factorization and Gated Projection (SFGP), which constructs a factorized semantic representation and selectively reweights its components for each target observer, constraining metric learning under limited triplet supervision. SFGP consistently improves target agreement and achieves larger observer-specific gains than conventional end-to-end triplet learning. Together, our results provide a framework for characterizing and evaluating similarity as a population of related yet individually distinct metrics, while showing that leveraging a shared factorized semantic representation can improve individual metric learning under limited observations.