Exploiting Anonymous Annotations for Cluster-Based Learning to Defer

Learning to defer (L2D) is a prominent paradigm in human-AI cooperation because it leverage both the efficiency of machine learning models and reliability of human experts. However, existing L2D methods are typically expert-specific, meaning that the same set of human experts annotating during training must also be present at deployment. Such a requirement limits the flexibility of L2D, particularly in dynamic real-world environments where expert availability can fluctuate due to leave, retirement, or the onboarding of new members. To accommodate such changes without redefining the router for every expert pool, we propose L2D-Clusters, a cluster-based L2D method that defers a query to a cluster, from which an expert (either a human or classifier) is selected to make the prediction. To construct these clusters while reducing the need for annotations linked to individual identities, we introduce a Bayesian mixed-membership model that learns reusable annotation-pattern components from anonymous annotations. The learnt model subsequently infers the cluster membership of a previously unseen expert using only a small identified annotation set without ground-truth labels. Experiments show that the learnt clusters can capture meaningful annotation clusters and that L2D-Clusters outperforms expert-specific and population-based baselines in the small identified data regime, while expert-specific methods regain their advantage when abundant identified data are available.