acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Anonymous Annotations for Cluster-Based Learning to Defer

Abstract

Learning to defer (L2D) is a prominent paradigm in human-AI cooperation because it leverage both the efficiency of machine learning models and reliability of human experts. However, existing L2D methods are typically expert-specific, meaning that the same set of human experts annotating during training must also be present at deployment. Such a requirement limits the flexibility of L2D, particularly in dynamic real-world environments where expert availability can fluctuate due to leave, retirement, or the onboarding of new members. To accommodate such changes without redefining the router for every expert pool, we propose L2D-Clusters, a cluster-based L2D method that defers a query to a cluster, from which an expert (either a human or classifier) is selected to make the prediction. To construct these clusters while reducing the need for annotations linked to individual identities, we introduce a Bayesian mixed-membership model that learns reusable annotation-pattern components from anonymous annotations. The learnt model subsequently infers the cluster membership of a previously unseen expert using only a small identified annotation set without ground-truth labels. Experiments show that the learnt clusters can capture meaningful annotation clusters and that L2D-Clusters outperforms expert-specific and population-based baselines in the small identified data regime, while expert-specific methods regain their advantage when abundant identified data are available.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
learning to defer, latent Dirichlet allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TXkAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes L2D-Clusters, which learns KK latent labeling components from anonymous annotations, infers an expert’s soft cluster membership from a small identified set without ground-truth labels, and routes each query first to a cluster and then to an available expert. The paper provides perturbation and variational-inference bounds and evaluates the method using synthetic experts on CIFAR-100 and dopanim and three pathologists on Chaoyang, reporting coverage–accuracy AUC, clustering ARI, departure experiments, and sensitivity analyses.

Critical strengths

Important practical problem. A cluster-indexed router whose output dimension does not depend on individual experts is a clean and potentially useful abstraction for workforce turnover and data-scarce onboarding.

Reasonably broad diagnostics. The paper studies soft and hard memberships, small- and large-context regimes, expert departure, 100 simulated experts, and hyperparameter sensitivity. It also candidly reports that expert-specific methods regain their advantage when sufficient identified data are available.

Critical weaknesses
  • The anonymous-clustering model is internally inconsistent and not identifiable under the stated construction. Section 3.1 draws a separate uiju_{ij} for each annotation and allows each SiS_i to mix annotators, whereas Appendix E derives a single uu shared by all annotations in SiS_i, and Algorithm 1 constructs SiS_i by random sampling. These are different models. More fundamentally, LDA can omit document authors but still requires meaningful document boundaries; arbitrary mixed-annotator minibatches provide no within-expert co-occurrence. With flexible neural components hkh_k, many decompositions—including collapsed components—can explain the same pooled conditional label distribution. Proposition 2 only bounds the variational posterior against the exact posterior of the assumed fitted model; it does not establish recovery of true expert structure or sample efficiency. This undermines the paper’s central interpretation of the components as reusable expert clusters.

  • The central unseen-expert/onboarding claim is not actually validated. Appendix N.2 describes a sample-level rather than annotator-level split, and the same synthetic experts appear to contribute to the anonymous pool before being assigned using held-out identified samples. The paper does not document a leave-one-expert-out experiment in which both the clusterer and router are frozen before a genuinely unseen expert is introduced; the explicit workforce-change experiment only removes known experts. Evidence is also predominantly synthetic. The sole real-expert dataset contains only three pathologists and uses their own majority vote as ground truth. At the selected K=3K=3, clustering ARI is only 0.440.44, tying IFD. Consequently, the experiments show reuse of behaviors already represented in pooled data, rather than reliable onboarding of a genuinely unseen expert.

Questions for authors
  • Which generative model is actually implemented, the per-annotation uiju_{ij} in Section 3.1 or the shared uiu_i in Appendix E, and what defines an SiS_i boundary? Under randomly formed mixed-annotator subsets, what assumptions make the component labeling functions identifiable?

  • Was any evaluation expert completely excluded from anonymous clusterer training, with both the clusterer and router frozen before that expert was assigned using only a small no-ground-truth set? If so, please provide the exact data chronology and results.

  • How many independent runs underlie the reported means and standard deviations, and how is coverage–accuracy AUC computed for methods having only one nontrivial coverage point?

7 comments
AuthorsOpenReview ↗

Which generative model is actually implemented

We thank the reviewer for this question because the description of the mixture coefficient u\mathbf{u} in Section 3.1 has a typo. The implemented model uses a single mixture ui\mathbf{u}_{i} shared by all annotations in the set Si\mathcal{S}_{i}. This shared model is the one shown in the graphical model (Figure 1c), derived in Appendix E (a single variational posterior q(u;ρ)q(\mathbf{u}; \rho) per set, where ρk=αk+∑n=1Niζnk\rho_{k} = \alpha_{k} + \sum_{n = 1}^{N_{i}} \zeta_{nk} pools all annotations in the set), and implemented in the code provided in the supplementary material. We have corrected Section 3.1 in the revision, with the changes highlighted in red and green.

AuthorsOpenReview ↗

What defines an boundary

In our setting, each Si\mathcal{S}_{i} consists of NiN_{i} annotated samples drawn at random from the anonymously annotated dataset. The model does not require any particular boundary for these sets. If Si\mathcal{S}_{i} contains annotations from several annotators with membership probability vectors ωm∈ΔK−1\omega_{m} \in \Delta^{K - 1} in proportions πm∈[0,1]\pi_{m} \in [0, 1], the annotation style of each annotation is drawn from ∑mπmωm\sum_{m} \pi_{m} \omega_{m}, which is itself a valid mixture ui\mathbf{u}_{i}. Mixing annotators within a set therefore does not violate the modelling assumptions.

AuthorsOpenReview ↗

Under randomly formed mixed-annotator subsets, what assumptions make the component labeling functions identifiable?

We do not claim identifiability in general. As with other latent variable models, including standard LDA, identifiability cannot be guaranteed without additional assumptions. The last paragraph in Section 3.1 (i.e., "Annotation diversity") states a necessary condition. Specifically, the anonymously annotated dataset must contain sufficiently many samples on which the annotation styles disagree. Otherwise, no method, including ours, can learn meaningful clusters. Sufficient conditions require further structural assumptions even for standard LDA, such as separability (Arora et al., 2012). Extending these assumptions to the current setting is beyond the scope of this paper. We therefore verify recovery empirically in a setting where the reference grouping is known: our method recovers the three pairs of experts exactly (ARI = 1) at K=5K = 5 (Table 2b) and partially (ARI = 0.44) at K=3K = 3 and K=4K = 4. In our revision, we have clarified this sufficient condition in the last paragraph of Section 3.1 as highlighted in green.

Additional references: Arora, S., Ge, R. and Moitra, A., 2012, October. Learning topic models--going beyond SVD. In 2012 IEEE 53rd annual symposium on foundations of computer science (pp. 1-10). IEEE.

Reviewer_3w7rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes L2D-clusters, a cluster-based learning-to-defer framework designed to accommodate changing pools of human experts. Instead of learning routing decisions for individual experts as in conventional expert-specific methods, the proposed method routes queries to clusters of experts and selects an individual within the chosen cluster to produce the final prediction. The method has two components: an LDA-inspired Bayesian mixed-membership model learns latent annotation patterns from anonymous annotation data, which are then used to infer the cluster membership of new experts, and a cluster-level router trained using an EM-based formulation with workload constraints. Experiments on CIFAR-100, dopanim, and Chaoyang demonstrate improved coverage-accuracy performance over several expert-specific and expert-agnostic baselines when identified annotation data are scarce.

Critical strengths
  1. The paper studies a practical limitation of existing L2D methods. Most methods assume the same experts are available during training and deployment, which may not always be true. The idea of learning annotation patterns from anonymous data and assigning new experts to clusters using only a small number of annotations without ground-truth labels is valuable.
  2. The authors evaluate the method on three datasets, including expert departure scenarios and a larger experiment with 100 synthetic experts. They also study the effect of the number of clusters and the Dirichlet prior and provide a way to select K using held-out data.
Critical weaknesses
  1. Most experiments use synthetic experts grouped by three error rates. It is unclear whether the proposed method learns anything beyond differences in annotator accuracy. The only real-expert evaluation uses three pathologists on Chaoyang, where the learned cluster memberships are nearly uniform (Fig. 7). It would be more convincing if the authors could evaluate the method on real annotators from datasets such as dopanim.
  2. The theoretical results mainly bound errors under estimated cluster assignments but do not explain when meaningful clusters can actually be recovered from anonymous annotations.
Questions for authors
  1. dopanim contains annotations from real annotators. Why did the authors use synthetic experts instead? It would be helpful to evaluate the method on real annotations from dopanim to see whether meaningful clusters can be learned.
  2. How many random seeds were used for Tables 1 and 2a? Are the improvements on Chaoyang statistically significant?
3 comments
AuthorsOpenReview ↗

Number of random seeds and the statistical significance on the dataset Chaoyang

We use 3 random seeds for every method. Each seed gives a complete coverage-accuracy curve, from which we compute one AUC with the trapezoidal rule. Tables 1 and 2a report the mean and standard deviation of these three AUCs. In the revision uploaded on Open Review, we have added this information in the caption of Table 1 and Table 2a.

For the Chaoyang dataset in Table 1, we perform a hypothesis test to determine if the improvements are statistically significant. Since each split corresponding to a random seed is evaluated across all methods, we use a paired t-test to evaluate the difference. We present the results of a paired t-test between each method and L2D-Clusters (hard), and report the two-tail p-values as follows (N′N^{\prime} denotes the number of annotator-identified samples):

MethodN′=50N^{\prime} = 50N′=100N^{\prime} = 100
One-stage0.2180.055
Two-stage0.1640.251
LECODU0.0540.0115
IFD0.00010.0001
L2D-Pop0.0890.042
L2D-Clusters (soft)0.2940.496

The improvement is significant against IFD, but not against the other baselines. This is because Chaoyang has only three pathologists and they are highly overlapped in terms of predictions. In addition, the ground truth is defined by the majority vote of these pathologists. As a result, deferring to either pathologist results in similar performance. In the revision, we have added the Appendix P to report the results of the statistical tests, and state that L2D-Clusters performs on par with the strongest baselines on Chaoyang.

AuthorsOpenReview ↗

Near-uniform membership on Chaoyang (Figure 7)

We thank the reviewer for this observation. The near-uniform memberships on Chaoyang are expected in our model, which reflects the difficulty of the identified data to separate the three pathologists. First, the three pathologists agree on most samples (pairwise agreement is 79% to 91% as mentioned in Appendix H). These highly overlapping annotations provide limited information to infer the cluster memberships of the three pathologists. Second, it is attributed to the modest value of the Dirichlet prior used in training to prevent cluster collapse when experts overlap heavily (see Section 3.1). Nevertheless, even though the results of our method in Figure 7 appear near uniform, they remain informative. Pathologists 1 and 2 lean towards opposite clusters since they agree least, while pathologist 3 who agrees on both, lies in between.

It is important to note that the high confidence of membership from existing baselines does not mean they are able to recover a clearer structure. The representation-based baselines (L2D-Pop and IFD) apply Fuzzy C-Means to the representations of the available experts, so the clusters are defined by the relative differences between these experts. When experts annotate similarly and should belong to one cluster, Fuzzy C-Means still separates them into different clusters with high confidence regardless of how similar they are. In contrast, our method assigns each expert individually. The membership of expert indexed by mm is inferred from that expert's identified annotations Im\mathcal{I}_{m} alone, according to how well each of the KK annotation styles learnt from the anonymous training data explains them (Section 3.2). It never compares experts with one another at test time. Thus, adding or removing an expert does not change the memberships of the others.

AuthorsOpenReview ↗

Real-world datasets

dopanim\mathsf{dopanim} is designed for multi-rater learning, not for L2D or expert clustering. We simulate the synthetic experts in dopanim\mathsf{dopanim} for two reasons. First, dopanim\mathsf{dopanim} provides no reference grouping of its human experts, so clustering quality cannot be quantified (e.g., by the Adjusted Rand Index in Table 2b). Second, it does not contain enough samples labelled by all human experts. Such samples are needed to evaluate deferral in L2D, because the selected expert's annotation must be available for every test sample. In contrast, simulated experts on dopanim\mathsf{dopanim} images can provide both of them.

We also note that, to the best of our knowledge, our work is the first to study the clustering of human experts from anonymously annotated data. Existing methods for grouping annotators (see the extended related work in Appendix O) require annotations linked to annotator identities, and no existing method or benchmark addresses the anonymous setting. For this reason, even the clustering baselines in Section 4.3 are not clustering methods by design. We had to adapt L2D-Pop and IFD by extracting their expert representations and clustering these representations with Fuzzy C-Means. More broadly, the scarcity of suitable real-world data is a known limitation of research on human-AI collaboration, and of L2D in particular. Evaluating deferral requires the selected expert's annotation for every test sample, and few real datasets provide this for more than a handful of experts. Consequently, most L2D studies evaluate on synthetic experts, either exclusively or alongside a small number of real-annotator datasets (e.g., Mozannar & Sontag, 2020; Verma & Nalisnick, 2022; Verma et al., 2023; Mao et al., 2023; Hemmer et al., 2023; Tailor et al., 2024; Zhang et al., 2024; Nguyen et al., 2025). Our evaluation follows this practice by using synthetic experts where a reference grouping is needed to quantify clustering quality. We also use the 3 real pathologists of Chaoyang to examine the behaviour of our method on real annotations (Appendix H).

Reviewer_DhZcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes L2D-Clusters, a cluster-based learning-to-defer framework designed to accommodate dynamic expert populations, where human experts may join or leave the system during deployment. Instead of learning to defer to individual experts, the proposed framework routes instances to clusters of experts and subsequently selects an expert within the chosen cluster. To construct these clusters, the authors develop a Bayesian mixed-membership model inspired by Latent Dirichlet Allocation (LDA), which learns shared annotation patterns from anonymous annotations. The learned model can then infer the cluster membership of a new expert using a small set of identified annotations without requiring ground-truth labels. The authors also provide theoretical analyses and conduct experiments to evaluate the proposed framework under limited-data, onboarding, and expert-departure settings.

Critical strengths
  1. The paper addresses a meaningful and practical limitation of existing learning-to-defer methods. Most conventional L2D methods rely on expert-specific information and assume a relatively stable expert population. The proposed cluster-based framework offers an alternative that can potentially accommodate expert onboarding and departure without rebuilding the entire deferral mechanism.
  2. The use of anonymous annotations is practically appealing. The proposed framework reduces the dependence on expert-specific annotation histories and does not require ground-truth labels when estimating the cluster membership of a newly introduced expert. This is particularly valuable in scenarios where collecting expert-specific annotations and verifying their correctness are expensive.
  3. The proposed framework is reasonably comprehensive. The paper integrates anonymous annotation modeling, expert clustering, and cluster-based deferral into a unified framework. It also provides theoretical analyses of the impact of clustering estimation errors on downstream deferral performance.
Critical weaknesses
  1. The novelty of the LDA-based clustering method is somewhat limited. The core clustering method is largely adapted from the classical LDA framework, where each expert is modeled as a mixture of latent annotation components. Therefore, the main contribution seems to lie primarily in combining existing techniques for a new L2D setting, rather than developing a fundamentally new clustering or deferral methodology. The authors should more clearly articulate the methodological advances beyond these established frameworks.
  2. The modeling assumptions underlying anonymous annotation clustering require further justification. The authors construct the anonymous annotation dataset by dividing annotations into subsets, each of which may contain annotations from either the same or different annotators. However, the proposed generative process appears to draw a separate Dirichlet-distributed mixture for each annotation rather than sharing a latent mixture across annotations associated with the same expert. This raises an important question about how the model can reliably recover expert-level annotation patterns when expert identities and annotation-level dependencies are unavailable during training.
  3. The experimental settings do not fully establish the generalizability of the proposed method. A substantial portion of the evaluation relies on synthetically generated experts with predefined annotation error rates. In particular, several clustering experiments use three pairs of experts whose similarities are intentionally constructed. While these experiments provide a controlled evaluation environment, they may not adequately capture the diversity and complexity of real-world expert behavior.
Questions for authors

Please see the Weaknesses.

3 comments
AuthorsOpenReview ↗

Novelty of the LDA-based clustering method

We agree that our clustering method is adapted from the Latent Dirichlet Allocation (LDA) model, as stated in Section 3.1, and we do not claim a new generic inference method for LDA. The main novelty of our paper is the formulation of the expert clustering problem from anonymously annotated data as a variant of LDA to work in the expert clustering setting.

The proposed adaptation is relevant because conventional LDA cannot be applied directly to the setting of anonymously annotated datasets. In standard LDA, each topic is a fixed multinomial distribution over a vocabulary. In our setting, a "word" is an annotation t\mathbf{t} conditioned on the corresponding input x\mathbf{x}, and an annotation function (or topic) must therefore describe an input-dependent distribution over labels. We, therefore, replace each fixed multinomial topic with a learnable input-conditional categorical distribution, $\Pr(\mathbf{t} | \mathbf{x}, z = k, \theta_{1:K}) = \mathrm{Categorical}(\mathbf{t} | h(\mathbf{x};\theta_{k})),$ where h(.;θk)h(.; \theta_{k}) represents a reusable annotation function indexed by kk with a parameter θk\theta_{k}. The resulting model jointly learns these annotation functions from anonymous annotations and represents each expert as a mixture over them, enabling the membership of a new expert to be inferred from a small identified annotation set without requiring ground-truth labels.

We agree that the novelty does not reside in LDA. Rather, it lies in:

  • formulating anonymous expert-pattern discovery as an input-conditional mixed-membership problem,
  • replacing conventional multinomial topics with learnable annotation functions h(x;θk)h(\mathbf{x}; \theta_{k}), and
  • connecting the resulting memberships to a cluster-indexed L2D router whose output dimension is independent of the identities and number of available experts.

We have added a paragraph at the end of Section~3.1 to make this scope and distinction explicit.

AuthorsOpenReview ↗

Generative model of anonymous annotation data

We thank the reviewer for spotting this. This is a typo in Section 3.1. In fact, the mixture for each "document" Si\mathcal{S}_{i} is shared across all annotations in that document. It is not drawn once per annotation. The rest of the submission already uses the shared model:

  • the graphical model in Figure 1c
  • the variational posterior in Appendix E, and in particular, the parameter ρ\rho of the posterior q(u∣ρ)q(\mathbf{u} | \rho) in Appendix E.2.2
  • the code in the supplementary material.

We have corrected the data generation process for the anonymous annotations in Section 3.1 (please see the revision uploaded to Open Review with the changes highlighted in red and green).

With the correction, annotations in each document Si\mathcal{S}_{i} share the same mixture ui\mathbf{u}_{i}. Training does not need expert identities because it does not try to recover individual experts. Instead, it learns the KK "annotation style topics" (or annotation functions) h(.;θk),k=1:Kh(.; \theta_{k}), k = 1:K. At test time, we infer the topic mixture u\mathbf{u}, and specifically its variational posterior, of a set I\mathcal{I} annotated by an expert to assign that expert into corresponding clusters.

AuthorsOpenReview ↗

Evaluation on synthetic and real-world datasets

We agree that most of our evaluation uses synthetic experts. This is a deliberate choice to measure the clustering quality, such as the Adjusted Rand Index in Table 2b. To our knowledge, no public dataset provides real annotators with their reference grouping in terms of annotation style. Without that information, it is not possible to quantify the clustering quality. That is why we leverage controlled simulations for the evaluation. Note that we follow the simulation settings in prior work (Hemmer et al., 2023; Nguyen et al., 2025) to use class- and instance-dependent label noise, which is closer to real annotation errors than the class-dependent noise commonly used in L2D benchmarks.

We also note that, to the best of our knowledge, our work is the first to study the clustering of human experts from anonymously annotated data to extend the capability of L2D. Many existing methods for clustering annotators (see the extended related work in Appendix O) require annotations linked to annotator identities, and no existing method or benchmark addresses the anonymous setting. For this reason, even the clustering baselines in Section 4.3 are not clustering methods by design. We had to adapt L2D-Pop and IFD by extracting their expert representations and clustering these representations with Fuzzy C-Means. More broadly, the scarcity of suitable real-world data is a known limitation of research on human-AI collaboration, and of L2D in particular. Evaluating clustering requires grouping reference, while evaluating deferral requires the selected expert's annotation for every test sample. Only few real datasets provide this for more than a handful of experts. Consequently, most L2D studies evaluate on synthetic experts, either exclusively or alongside a small number of real-annotator datasets (e.g., Mozannar & Sontag, 2020; Verma & Nalisnick, 2022; Verma et al., 2023; Mao et al., 2023; Hemmer et al., 2023; Tailor et al., 2024; Zhang et al., 2024; Nguyen et al., 2025). Our evaluation follows this practice, where we use synthetic experts where a reference grouping is needed to quantify clustering quality, and the three real pathologists of Chaoyang to examine the behaviour of our method on real annotations (Appendix H).

Comments on the paper

1 comment

Dear reviewers DhZc, 3w7r and TXkA,

As the deadline of the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript, and let us know if you have any remaining concerns. We would be happy to provide further clarification if needed.

Kind regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.