- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes L2D-Clusters, which learns latent labeling components from anonymous annotations, infers an expert’s soft cluster membership from a small identified set without ground-truth labels, and routes each query first to a cluster and then to an available expert. The paper provides perturbation and variational-inference bounds and evaluates the method using synthetic experts on CIFAR-100 and dopanim and three pathologists on Chaoyang, reporting coverage–accuracy AUC, clustering ARI, departure experiments, and sensitivity analyses.
Critical strengths
Important practical problem. A cluster-indexed router whose output dimension does not depend on individual experts is a clean and potentially useful abstraction for workforce turnover and data-scarce onboarding.
Reasonably broad diagnostics. The paper studies soft and hard memberships, small- and large-context regimes, expert departure, 100 simulated experts, and hyperparameter sensitivity. It also candidly reports that expert-specific methods regain their advantage when sufficient identified data are available.
Critical weaknesses
-
The anonymous-clustering model is internally inconsistent and not identifiable under the stated construction. Section 3.1 draws a separate for each annotation and allows each to mix annotators, whereas Appendix E derives a single shared by all annotations in , and Algorithm 1 constructs by random sampling. These are different models. More fundamentally, LDA can omit document authors but still requires meaningful document boundaries; arbitrary mixed-annotator minibatches provide no within-expert co-occurrence. With flexible neural components , many decompositions—including collapsed components—can explain the same pooled conditional label distribution. Proposition 2 only bounds the variational posterior against the exact posterior of the assumed fitted model; it does not establish recovery of true expert structure or sample efficiency. This undermines the paper’s central interpretation of the components as reusable expert clusters.
-
The central unseen-expert/onboarding claim is not actually validated. Appendix N.2 describes a sample-level rather than annotator-level split, and the same synthetic experts appear to contribute to the anonymous pool before being assigned using held-out identified samples. The paper does not document a leave-one-expert-out experiment in which both the clusterer and router are frozen before a genuinely unseen expert is introduced; the explicit workforce-change experiment only removes known experts. Evidence is also predominantly synthetic. The sole real-expert dataset contains only three pathologists and uses their own majority vote as ground truth. At the selected , clustering ARI is only , tying IFD. Consequently, the experiments show reuse of behaviors already represented in pooled data, rather than reliable onboarding of a genuinely unseen expert.
Questions for authors
-
Which generative model is actually implemented, the per-annotation in Section 3.1 or the shared in Appendix E, and what defines an boundary? Under randomly formed mixed-annotator subsets, what assumptions make the component labeling functions identifiable?
-
Was any evaluation expert completely excluded from anonymous clusterer training, with both the clusterer and router frozen before that expert was assigned using only a small no-ground-truth set? If so, please provide the exact data chronology and results.
-
How many independent runs underlie the reported means and standard deviations, and how is coverage–accuracy AUC computed for methods having only one nontrivial coverage point?
7 comments
What defines an boundary
In our setting, each consists of annotated samples drawn at random from the anonymously annotated dataset. The model does not require any particular boundary for these sets. If contains annotations from several annotators with membership probability vectors in proportions , the annotation style of each annotation is drawn from , which is itself a valid mixture . Mixing annotators within a set therefore does not violate the modelling assumptions.
Under randomly formed mixed-annotator subsets, what assumptions make the component labeling functions identifiable?
We do not claim identifiability in general. As with other latent variable models, including standard LDA, identifiability cannot be guaranteed without additional assumptions. The last paragraph in Section 3.1 (i.e., "Annotation diversity") states a necessary condition. Specifically, the anonymously annotated dataset must contain sufficiently many samples on which the annotation styles disagree. Otherwise, no method, including ours, can learn meaningful clusters. Sufficient conditions require further structural assumptions even for standard LDA, such as separability (Arora et al., 2012). Extending these assumptions to the current setting is beyond the scope of this paper. We therefore verify recovery empirically in a setting where the reference grouping is known: our method recovers the three pairs of experts exactly (ARI = 1) at (Table 2b) and partially (ARI = 0.44) at and . In our revision, we have clarified this sufficient condition in the last paragraph of Section 3.1 as highlighted in green.
Additional references: Arora, S., Ge, R. and Moitra, A., 2012, October. Learning topic models--going beyond SVD. In 2012 IEEE 53rd annual symposium on foundations of computer science (pp. 1-10). IEEE.
Which generative model is actually implemented
We thank the reviewer for this question because the description of the mixture coefficient u in Section 3.1 has a typo. The implemented model uses a single mixture ui shared by all annotations in the set Si. This shared model is the one shown in the graphical model (Figure 1c), derived in Appendix E (a single variational posterior q(u;ρ) per set, where ρk=αk+∑n=1Niζnk pools all annotations in the set), and implemented in the code provided in the supplementary material. We have corrected Section 3.1 in the revision, with the changes highlighted in red and green.