acceptodds
Under review as a conference paper at ICLR 2027

The Same-Family Halo: A Gold-Free Audit of Source-Dependent Agreement in LLM Silver Labeling

Abstract

Scalable analysis of long-form human–human, human–agent, and agent–agent interactions requires reliable supervision. Large language models (LLMs) provide a practical source of silver labels, but agreement among models can reflect shared labeling preferences rather than independent confirmation. We investigate this dependence in an enterprise pipeline that uses 76,000 silver-labeled customerservice interactions to train classifiers operating over tens of millions of conversations, with a separate human-labeled holdout of 924 examples. We introduce a gold-free audit: holding each labeler’s predictions fixed, we vary the model supplying the reference labels and measure changes in agreement without consulting human annotations. Across 48 model–prompt–rendering configurations, we observe source-dependent agreement that extends beyond exact self-comparisons to sibling models. In a representative comparison, agreement with sibling-model labels exceeds agreement with three cross-family sources by 3.7–3.9 percentage points. To mitigate this effect, we propose combinatorial silver-label construction, assigning each example to a randomly selected model–prompt tuple — randomizing the prompt as well, since prompt choice is itself a first-order driver of label quality. Under this construction, the source-specific agreement advantage seen with fixed-source labels is directionally reduced in our evaluation, though not to statistical significance at our sample size. The method distributes supervision across configurations while preserving exactly one inference call per example. Our central — and demonstrated — finding is that same-family consensus can reflect a source-dependent agreement pattern that resembles independent confirmation while not providing it. The gold-free audit makes this dependence measurable even where human reference labels are scarce, and randomized construction offers a practical, single-call route toward mitigating it in scalable conversation analytics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, silver labeling, model–prompt bias, randomized annotation, weak supervision, conversation analytics, scalable classification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jak6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether agreement among LLM-generated silver labels provides independent evidence of label reliability. It introduces a gold-free audit that fixes the evaluated labeler while varying the model supplying the silver reference, and reports a “same-family halo”: models tend to agree more strongly with labels produced by related model families than with cross-family sources. The study evaluates a broad model–prompt configuration grid on a fine-grained enterprise classification task and additionally analyzes correlated errors and effective panel diversity. It further proposes randomizing the model–prompt source of silver labels per example as a low-cost mitigation. The empirical evidence for the halo itself is reasonably strong, while the evidence for the proposed mitigation is explicitly preliminary and not statistically conclusive.

Critical strengths
  1. A clean and practically relevant diagnostic. Holding the labeler fixed while changing only the silver-label source is a simple but effective way to expose source-dependent agreement without relying on potentially imperfect human gold. This is the most compelling contribution and makes the result relevant to real LLM-labeling and weak-supervision pipelines.

  2. Careful experimental framing and statistical discipline. The paper evaluates many model/prompt configurations, uses paired comparisons and multiplicity corrections, and generally avoids overstating results. In particular, the authors clearly separate the demonstrated halo from weaker evidence concerning mitigation.

Critical weaknesses
  1. The central family-specific interpretation remains underidentified. The experiment demonstrates source-dependent agreement convincingly, but it does not cleanly establish that family membership itself is the important organizing variable rather than capability similarity, architectural/output similarity, or other correlated properties. This matters because the family interpretation is what makes the main practical recommendation distinctive relative to existing work on correlated LLM errors and self-preference.

  2. The proposed mitigation and broader practical conclusion are not yet sufficiently validated. The randomized silver-label construction produces only directional improvements whose confidence intervals include zero, and the paper does not demonstrate that the proposed strategy improves either silver-label correctness or downstream classifier performance. The evidence is also restricted to one proprietary application domain, substantially limiting confidence in the generality of the prescription.

Questions for authors

Can the authors provide a capability-matched analysis of same-family versus cross-family sources—for example by matching or controlling for individual label accuracy or another measure of model capability? A persistent halo after such matching would substantially strengthen the central interpretation.

Can the authors provide either an independent-dataset replication or substantially broader evidence involving multiple genuinely distinct models within several families? This would help determine whether the reported phenomenon is a general family-level effect rather than a property of a small number of particular model pairs.

Is there evidence that randomized silver construction ultimately improves label correctness or downstream model performance, rather than only reducing the proposed family-dispersion statistic? A positive result here would materially change my assessment of the practical contribution.

Reviewer_4h33OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether agreement among LLM labelers is a signal of quality. Using 924 human-labeled customer-service calls. They report that labelers agree more with labels from same-family models than from other families (about +3.8pp for Haiku against Sonnet versus three cross-family sources).

Critical strengths

The setting (real production data, a large fine-grained taxonomy) is practically relevant.

Critical weaknesses

W1. The paper is hard to follow. The task is never properly introduced, background is assumed, and the text relies on many terms that are defined late, loosely, or not at all. In the abstract alone: silver labels (L013), gold-free audit (L019), model–prompt–rendering configuration (L021). Fig. 1 does not clarify the method either. Even the main claim of the paper is hard to parse: "Our central --- and demonstrated --- finding is that same-family consensus can reflect a source-dependent agreement pattern that resembles independent confirmation while not providing it."(L031) It is unclear what “same-family consensus”, “source-dependent agreement pattern” and “independent confirmation” refer to. I would encourage the authors to restate the whole paper in plain terms and introduce the task and terminology before using them. I will be happy to review the revised paper then.

W2. Among what I could understand I'm still confused by some claims: the abstract states that the proposed construction reduces the source-specific agreement advantage, while also stating that the reduction is not statistically significant (Table 4, all CIs include zero). A non-significant reduction should not be presented as an effect, and the randomized construction should not be listed as a contribution on this basis.

W3. It seems that the main finding is tied to models in the same family agreeing on the output label, which seems to be a well known issue of LLMs.

Questions for authors

The novelty relative to prior work is unclear to me. The “same-family halo” is pairwise agreement between independently produced labels, i.e. correlated errors between sibling models, which is the phenomenon studied by Kim et al. (2025)[1]. Could the authors explain how does this work differs?


[1] Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. Correlated errors in large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 1178, 30038–30066.

Reviewer_nST6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the dependence of machine generated “silver labels” across different estimators and against a human created “gold” source. They propose to use randomization to reduce the observed intra-family correlation.

Critical strengths

The paper’s topic, generalizing a small number of strong labels to a large set of cheap, weak labels is a quite common and important problem for machine learning

Critical weaknesses

The paper has no comparison to other techniques designed to reduce correlation across an ensemble of classifiers, which is a quite standard and well known problem (especially in NNs where “ensemble heads, shared trunk” designs are not uncommon). There are several well known and established techniques that should be compared against: Covariance-aware ensemble weighting is probably the closest formal analog going back to Bates and Granger (https://www.tandfonline.com/doi/abs/10.1057/jors.1969.103): Estimate the covariance of the members’ errors on validation data (in your case gold labels) and choose weights accounting for that covariance. For classification, the direct element would be Brier/probability residuals (see also https://www.sciencedirect.com/science/article/pii/S1566253504000375). Effectively you compute the errors on your hold out (gold) set, build a covariance matrix and then define the weightings as min⁡wwTΣw\min_w w^T\Sigma w

Stacking is another common option: Instead of explicitly computing correlations, just optimize minw−∑nlog⁡(∑iwipi(yn∣xn))min_w -\sum_n \log\left( \sum_i w_i p_i(y_n|x_n)\right) s.t. wi≥0,∑iwi=1w_i\geq 0, \sum_iw_i=1. In words: you keep the classifiers fixed, but estimate the maximum likelihood weights wiw_i on your hold out set. (see https://arxiv.org/abs/1704.02030) In fact, since you only have a tiny number of estimators, you could even perform full Bayesian inference on wiw_i to give you uncertainty there as well. Maybe also have a look at https://www.tandfonline.com/doi/abs/10.1080/095400996116839 which looks at precisely the “many, perhaps slightly correlated estimator” setting. These are the bare minimum I would consider since these are textbook approaches that are proven to work (both in theory and in practice) for precisely the setting you are considering.

The proposed randomized method is in "ensemble-speak" simply sampling from the posterior introduced by the different models in the ensemble, i.e. instead of marginalizing out the choice of model ∑ip(y∣x,θi)q(θi)\sum_i p(y|x,\theta_i)q(\theta_i) you sample a model from q(θ)q(\theta), apply it, and then build the prediction from that. In fact one can potentially do better than fully random by sampling proportionally to some relevance density q(θ)q(\theta) instead of uniformly as done in this work. This is not in and off itself novel.

The paper would also benefit from making their claims more precise: Just sharing data does not mean these models are not iid, but rather that they are conditionally independent when conditioned on the data. This is the same behavior as in normal ensembles, which themselves are also trained on, at the very least, large overlaps of the training set. For ensembles, one typically assumes conditional independence given the data, not full independence.

The study is also way too small to make any reasonable inferences: Only 4 families and no NxN comparison matrix between in- and out- of family models. I think to make any reasonable inferences, one would need to look at more and larger families e.g. the Qwen models, which both offer a large number of model sizes, and, due to their open-weight nature, many finetunes like the ornith family.

Since these models can be run locally, this may also reduce the PII issues of sending this data, even pseudonymized to an external vendor.

Questions for authors
  • What is the performance of alternative correlation techniques
  • What is the performance inside larger families or finetuned derivativs
  • Can you produce a full NxN covariance between the different models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.