acceptodds
Under review as a conference paper at ICLR 2027

Observer-Specific Metric Learning via Semantic Factorization and Gated Projection

Abstract

Similarity judgments can vary across observers: observers may emphasize different attributes of the same data and thereby induce distinct similarity metrics. Yet conventional metric learning does not explicitly distinguish observer-specific structure from similarity shared across a population. We address two aspects of this problem. First, we formulate observer-specific metric learning (OSML) as recovering a target observer's similarity metric from sparse relational judgments. To distinguish observer-specific learning from population-shared predictability, we introduce a population-aware evaluation framework and Observer-Specific Gain (OSG), which compares agreement with the target observer against agreement with non-target observers. Across datasets and neural-network observer populations, population diversity relates to cross-observer predictability, supporting the need for population-aware evaluation. Second, we propose Semantic Factorization and Gated Projection (SFGP), which constructs a factorized semantic representation and selectively reweights its components for each target observer, constraining metric learning under limited triplet supervision. SFGP consistently improves target agreement and achieves larger observer-specific gains than conventional end-to-end triplet learning. Together, our results provide a framework for characterizing and evaluating similarity as a population of related yet individually distinct metrics, while showing that leveraging a shared factorized semantic representation can improve individual metric learning under limited observations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Metric Learning, Personalized Representation Learning, Semantic Factorization, Population-Aware Metric

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XPXWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to recover an observer's similarity metric from sparse triplet judgments. It introduces Observer-Specific Gain (OSG) to measure target agreement relative to agreement with other observers. Its proposed Semantic Factorization and Gated Projection (SFGP) uses self-supervised semantic factorization followed by observer-specific gated projection. Experiments with neural-network observers on CUB, Stanford Cars, and ImageNet-100 show improvements over direct metric learning with limited supervision.

Critical strengths
  • Population-relative evaluation makes a useful distinction between matching a target observer and reproducing judgments shared across observers.

  • SFGP improves target agreement over direct metric learning in several settings, and the component ablations help explain its design.

Critical weaknesses
  • The practical motivation remains unclear. A concrete use case would help explain why an observer-specific similarity metric is needed and what the proposed formulation offers beyond existing personalization or conditional metric learning approaches. Conditional Similarity Networks (Veit et al., 2017) and SCE-Net (Tan et al., 2019) already study different similarity criteria, such as color or category. Although their supervision differs, clearer positioning and comparisons under compatible access assumptions are needed.

  • SFGP is evaluated with only 1,000 or 2,000 training judgments per observer. The gains may primarily reflect better regularization and data efficiency rather than a distinct advantage in recovering observer-specific structure. Data efficiency is valuable, but learning curves extending to 30,000 triplets or more would clarify whether a well-tuned direct triplet baseline catches up.

  • The observers are trained on the same classes with the same objective and differ mainly in sampled images and random seeds. It is unclear whether their disagreement reflects different similarity preferences or ordinary training variation. Testing observers trained on different class sets, or trained to judge similarity by color versus category, would better show whether SFGP captures meaningful differences between observers.

  • Generalization to new images and classes is not clearly established. The paper separates triplets but does not specify whether images overlap across the splits. For the same target observer, training the metric learner on 100 CUB classes and testing on the remaining 100 would test whether its learned metric transfers beyond familiar classes. Test images and classes should be excluded from task-specific training, including factorizer pretraining. Evaluation under domain shifts would also help establish broader generalization.

Questions for authors
  • What concrete application requires an observer-specific similarity metric? How does SFGP differ from CSN?

  • Does SFGP retain its advantage over direct metric learning with substantially more training triplets?

  • Does SFGP capture differences between observers trained on different class sets or trained to judge similarity by different attributes, such as color or category?

  • Are the metric learner's training, validation, and test images disjoint? For a fixed target observer, does the learned metric generalize to held-out classes and domain shifts?

Reviewer_i1DJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to learn a similarity metric that reflects an individual observer’s preferences from a limited number of triplet similarity judgments, formulating the task as Observer-Specific Metric Learning (OSML). The paper has two main components. First, it introduces a population baseline and Observer-Specific Gain (OSG) to evaluate a learner’s agreement with the target observer relative to its agreement with other observers. Second, it proposes Semantic Factorization and Gated Projection (SFGP), which first constructs a shared representation through factorization pretraining, then freezes that representation and learns observer-specific gates and a metric head. Experiments on multiple image datasets and populations of independently trained neural-network observers show improvements in target agreement in most configurations, with improvements in OSG in some settings.

Critical strengths
  1. The paper addresses a meaningful evaluation issue in individual similarity learning The paper highlights that high agreement with a target observer does not necessarily imply recovery of the preferences that distinguish that observer from others, because some predictive accuracy may arise from similarity structure shared across the population. Evaluating both target agreement and cross-observer agreement encourages a more careful interpretation of personalization results. This evaluation perspective is one of the paper’s more distinctive contributions.
  2. The method is reasonably motivated by the challenges of sparse supervision SFGP separates shared representation construction from observer-specific adaptation. It first performs factorization pretraining without observer judgments, then uses limited triplet supervision to learn gates and a metric head. This design introduces a structural prior for learning from sparse relational data and supports reuse of the shared representation. The compression, expansion, recombination-consistency, and gating components form a coherent implementation of this approach.
  3. The experiments cover multiple settings and provide preliminary evidence of effectiveness The evaluation includes multiple datasets, different backbones, several metric-learning losses, and different supervision budgets, together with conventional autoencoder and β-VAE baselines and component ablations. Improvements in target agreement across most configurations suggest practical value and indicate that the benefits are not restricted to a single loss function. However, these results primarily support the effectiveness of the overall pipeline; the independent contributions of its proposed mechanisms require further validation.
Critical weaknesses
  1. The methodological novelty is insufficiently distinguished from closely related prior work Several central ideas have direct precedents: shared embeddings with conditional masks in Conditional Similarity Networks (CSN) [1], shared semantic views with observer-dependent preferences in crowdsourced multi-view triplet learning [2], and latent mixing with cycle-based constraints for factorization [3]. The paper lacks a systematic comparison with these approaches and therefore does not sufficiently establish what additional capability its particular combination provides. The current experiments primarily compare different loss functions, conventional autoencoding, and one β-VAE configuration. They do not adequately establish an advantage over the closest existing approaches, with suitable adaptations to respect the paper’s supervision protocol.
  2. The experimental controls do not sufficiently isolate the contributions of the proposed mechanisms A conventional autoencoder already produces substantial improvements in some settings, while SFGP’s additional gains are comparatively limited. Moreover, the ablations do not cleanly isolate key mechanisms. For example, removing LpermL_{\mathrm{perm}} eliminates both ordinary latent-cycle consistency and consistency after latent recombination, making it difficult to determine whether cross-sample channel recombination provides an independent benefit. The gating ablation likewise does not adequately separate the effects of importance-based initialization, additional gate parameters, metric-head reinitialization, and extra training. The evidence is stronger for the effectiveness of the complete pipeline than for the claimed roles of its individual innovations.
  3. The interpretation and validation of OSG remain limited The population baseline in OSG is the same learner’s average agreement with other observers, rather than an independently estimated population-shared component. OSG can therefore describe a relative agreement advantage, but it should not be directly interpreted as the amount of observer-specific information recovered. OSG is also affected by the composition and disagreement level of the reference population. The paper lacks controlled experiments with known individual preferences that would establish whether the metric reliably distinguishes recovery of shared structure from recovery of observer-specific deviations.
  4. Evidence for semantic factorization and recovery of meaningful individual preferences is limited Reduced channel correlations and selected activation maps do not establish that latent channels consistently represent interpretable, compositional semantic factors. They also do not demonstrate that the learned gates accurately reflect an observer’s semantic preferences. Furthermore, all experimental observers are independently trained neural networks, whose differences primarily arise from training stochasticity and differences in training samples. This setting is useful for studying recovery of model-specific metrics, but it does not yet establish that the conclusions extend to stable human preferences or their semantic interpretation.
Questions for authors
  1. What is SFGP’s substantive methodological advance over the shared-embedding masks in CSN [1], observer-preference modeling in Lu et al. [2], and latent mixing in Hu et al. [3], once differences in supervision protocols are taken into account?
  2. Under matched architectures and total training budgets, which gains specifically require shuffled latent-cycle training and permutation-based gate initialization, beyond ordinary autoencoding, an unshuffled latent cycle, and a uniformly initialized trainable gate?
  3. Can controlled experiments with known shared and observer-specific components establish that OSG tracks recovery of individual deviations while controlling for the diversity and composition of the reference population?
  4. What quantitative evidence supports interpreting the learned channels and gates as reflecting stable observer-specific semantic preferences, particularly when extending conclusions from neural-network observers to humans?
Reviewer_kiBGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Observer-Specific Metric Learning (OSML) which recover a single target observer’s similarity metric using only a sparse set of observer’s triplet judgments. It proposes Observer-Specific Gain (OSG), a population-relative evaluation metric, and Semantic Factorization and Gated Projection (SFGP), which learns a shared factorized representation and selectively reweights its components for each observer.

Critical strengths

The technique and novelty of this paper is sound. The OSG addresses the limitation of conventional metric learning by distinguishing target agreement from agreement shared across observers.

The experimental results are promising: SFGP improves target agreement in 58 of 60 reported settings, with substantial gains under sparse triplet supervision. Experiments cover multiple datasets, backbone architectures, learning objectives, and supervision budgets, supporting the method's applicability across different settings.

The authors provide detailed implementation settings and acknowledge limitations concerning OSG interpretation and semantic disentanglement.

Critical weaknesses

Incomplete baseline controls: The comparison does not clearly isolate representation factorization from differences in parameter freezing, model capacity, and training procedures. Image-level overlap between training and evaluation triplets is also unclear.

Insufficient evidence for semantic factorization and gating: Reduced feature correlation and localized activation maps do not establish meaningful semantic factorization.

Tables 3 and 4 report different target accuracies for the same ViT-Small/ImageNet-100 configuration (SFGP).

Questions for authors

Can OSG be validated against population-consensus predictors and controlled scenarios with known observer-specific factors?

Can the authors provide statistical uncertainty estimates and resolve the numerical inconsistencies across tables?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.