acceptodds
Under review as a conference paper at ICLR 2027

CALE: Cost-Aware Large-small Ensemble for Heterogeneous LLM Annotation Pools

Abstract

Large language models (LLMs) are increasingly used as annotators, and organizations often have access to heterogeneous pools of large and small LLMs with different costs and capabilities. We study the practical yet underexplored setting of annotating a fixed pool of unlabeled instances under a finite budget, where both labels and model reliability are initially unknown. The goal is to allocate this budget across models and instances to obtain high-quality annotations. We propose CALE, a cost-aware large–small ensemble framework that couples label aggregation with sequential acquisition. CALE models heterogeneous annotator reliability hierarchically, sharing information across similar instances while preserving instance-specific variation, and treats repeated calls to the same model as dependent rather than independent evidence. It then selects each model–instance query using complementary gains from improving the current label and learning reliability for future decisions, normalized by query cost. The resulting closed-loop interaction allows aggregation and acquisition to reinforce each other throughout the annotation process. Across five benchmarks spanning topic classification, clinical inference, entailment, ontology classification, and multi-subject question answering, annotated by heterogeneous LLM pools, CALE achieves a higher area under the accuracy–cost curve (AUBC) than every other baseline on every dataset, with the largest gains at tight budgets: it reaches 85.5% mean final accuracy at 1.5N, whereas competing systems require about 2.9N to reach the same level. Ablation and robustness studies analyze the sources of the gains and assess stability across modeling and deployment choices.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM annotation, multi-LLM collaboration, Bayesian crowdsourcing, cost-aware LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pxNLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CALE combines hierarchical label aggregation with cost-aware acquisition to annotate a fixed dataset using heterogeneous LLMs under a shared budget. Building on confusion-matrix aggregation, it models regime-specific reliability and dependence among repeated calls to the same model on the same example. Query selection balances expected label improvement and a surrogate for learning reliability against query cost. The experiments cover five text-classification and multiple-choice benchmarks.

Critical strengths
  1. Promising empirical performance. The method achieves the highest reported area under the accuracy–cost curve on all five benchmarks, supporting its usefulness for budget-constrained annotation (Table 1, p. 7).
  2. Relevant modeling choices. The framework incorporates regime-specific reliability, repeated-call dependence, and heterogeneous query costs into a coupled acquisition and aggregation procedure.
  3. Component analyses. The hierarchy, acquisition, and aggregation ablations help assess the contributions of the method’s components (Tables 2–4, p. 8), complemented by sensitivity analyses in Appendix G.
Critical weaknesses
  1. The incremental benefit of the proposed aggregation needs stronger evidence.
    With the acquisition history held fixed, Table 4 reports average final accuracy of 88.43% for CALE, compared with 87.92% for majority vote and 87.82% for Dawid–Skene. These improvements are modest, and the table does not report significance tests or confidence intervals for the paired differences. The corresponding average AUBC gains are also small: 0.20 and 0.15 points, respectively (Table 10, p. 18).

    This does not establish that the hierarchical aggregator is unnecessary, since it also guides acquisition. However, stronger evidence is needed to distinguish its contribution to final label aggregation from its contribution to query selection, and to justify the added complexity.

  2. Robustness to correlated errors across models remains unclear.
    CALE explicitly models dependence among repeated calls to the same model, while regimes can capture some shared difficulty patterns. However, residual cross-model dependence remains unmodeled, as acknowledged in Appendix B (p. 14). Models sharing systematic mistakes could produce misleading agreement and overconfident label estimates.

    This limitation is also shared by standard Dawid–Skene, so my concern is whether CALE retains its empirical advantage over the baselines under strongly correlated model errors. A controlled analysis would help establish the scope of its dependence-aware formulation.

Questions for authors
  1. Are the aggregation improvements in Table 4 statistically significant under a paired analysis across the shared acquisition histories? Can you clarify how much the hierarchical aggregator contributes through query selection versus final label aggregation?
  2. How does CALE perform relative to the baselines when different models have strongly correlated errors, such as repeatedly making the same mistakes?
2 comments

We sincerely thank the reviewer for the detailed and thoughtful evaluation. Below we provide detailed point-by-point responses.


Weakness1: The incremental benefit of the proposed aggregation needs stronger evidence.

Same as Q1: Are the aggregation improvements in Table 4 statistically significant under a paired analysis across the shared acquisition histories? Can you clarify how much the hierarchical aggregator contributes through query selection versus final label aggregation?

We thank the reviewer for highlighting the need for paired statistical evidence and a clearer explanation of the aggregator's contribution.

Final aggregation on shared acquisition histories. We compared CALE with alternative aggregators on the same 50 dataset--seed acquisition histories. The paired differences are:

ComparisonΔ\Delta Final Accuracy (95% CI)Holm-corrected ppΔ\Delta AUBC (95% CI)Holm-corrected pp
CALE −- MV+0.51 [0.42, 0.61]8.47×10−68.47\times10^{-6}+0.20 [0.17, 0.23]1.11×10−81.11\times10^{-8}
CALE −- DS+0.61 [0.49, 0.71]3.61×10−53.61\times10^{-5}+0.14 [0.12, 0.17]0.0149
CALE −- GLAD+0.60 [0.49, 0.72]2.11×10−42.11\times10^{-4}+0.14 [0.11, 0.17]2.61×10−42.61\times10^{-4}

The confidence intervals use paired bootstrap resampling within each dataset, and the pp-values use a two-sided paired Wilcoxon test with Holm correction. Thus, although the average gains are modest, the final aggregation improvement is statistically supported when the evidence is held fixed.

Interaction with query selection. Because aggregation and acquisition form a closed loop in the full method, the total improvement cannot be uniquely or additively decomposed into an “aggregation percentage” and an “acquisition percentage.” The fixed-history analysis above isolates the conditional benefit of final aggregation, while the end-to-end improvement reflects their joint interaction.


Weakness2: Robustness to correlated errors across models remains unclear.

Same as Q2: How does CALE perform relative to the baselines when different models have strongly correlated errors, such as repeatedly making the same mistakes?

We appreciate this important observation. We examine it through a shared-confusion extension of CALE and a stress test of the original CALE with deliberately correlated responses, both using the paper's original datasets and evaluation protocol. AUBC (%) is reported as mean ±\pm sample standard deviation.

Experiment 1: Shared-error modeling. We extended CALE with an instance-level latent state HiH_i shared by all models:

Hi∣Yi=y,Ki=k∼Cat⁡(Sk,y,:).H_i \mid Y_i=y,K_i=k \sim \operatorname{Cat}(S_{k,y,:}).

ϕig∣Hi=h,Ki=k∼Dir⁡(κTg,k,h,:).\phi_{ig}\mid H_i=h,K_i=k \sim \operatorname{Dir}(\kappa T_{g,k,h,:}).

Aigv∣ϕig∼Cat⁡(ϕig).A_{igv}\mid\phi_{ig} \sim \operatorname{Cat}(\phi_{ig}).

Here, SkS_k is the shared confusion matrix and Tg,kT_{g,k} is the model-specific response matrix, giving the marginal confusion Ψg,k=SkTg,k\Psi_{g,k}=S_kT_{g,k}. Marginalizing HiH_i induces cross-model dependence, while the shared ϕig\phi_{ig} retains dependence between repeated responses from the same model.

MethodAG NewsMedNLIRTEMMLUDBpedia-14Avg.
Shared-base78.75 ± 0.2074.71 ± 0.4485.45 ± 0.7271.67 ± 0.4484.96 ± 0.1279.11 ± 0.16
CALE78.83 ± 0.2474.48 ± 0.2686.02 ± 0.6571.30 ± 0.2284.90 ± 0.1279.11 ± 0.14

Explicitly modeling systematic errors shared across models does not provide a clear AUBC advantage on these datasets, while adding one C×CC\times C matrix per regime, or KC(C−1)K C(C-1) additional free parameters, thereby increasing model complexity.

Experiment 2: Correlated-error stress test. We introduce ϵ\epsilon as the probability of fully copying one model's complete three-response label sequence to another model for the same item. A larger ϵ\epsilon creates stronger repetition and cross-model dependence. We study ϵ=0.6\epsilon=0.6. AUBC (%) results:

MethodAG NewsMedNLIRTEMMLUDBpedia-14Avg.
DLTA75.98 ±\pm 0.1173.53 ±\pm 0.1284.22 ±\pm 0.5770.56 ±\pm 0.9885.19 ±\pm 0.1677.90 ±\pm 0.28
LMCrowd75.77 ±\pm 0.1873.94 ±\pm 0.3184.65 ±\pm 0.2169.65 ±\pm 0.5085.57 ±\pm 0.1477.92 ±\pm 0.09
CALE76.57 ±\pm 0.2373.78 ±\pm 0.2684.32 ±\pm 0.1270.02 ±\pm 0.8285.36 ±\pm 0.1378.01 ±\pm 0.16

CALE achieves the highest average AUBC, outperforming both DLTA and LMCrowd in this setting. This example shows that CALE can maintain a performance advantage when responses across models become substantially repetitive.


We hope that our responses have adequately addressed the concerns raised. We respectfully ask if you would be comfortable reconsidering the rating in light of these efforts, and we remain happy to address any further questions.

Thank you for the detailed rebuttal and additional experiments. The paired statistical analysis adequately addresses my concern about the significance of aggregation improvements, although the effect sizes remain modest. The correlated-error experiments also provide useful evidence of robustness, despite relatively small gains over baselines.

Overall, the new evidence addresses my main concerns and strengthens the paper. I am updating my rating from 2 (Weak Reject) to 3 (Weak Accept).

Reviewer_DXLROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CALE, which allocates a fixed annotation budget across LLMs with different costs and capabilities. It combines hierarchical reliability estimation with cost-aware query selection, accounting for dependence between repeated calls to the same model. It provides useful experiments on five classification benchmarks that show improved annotation accuracy per unit cost without using gold labels during execution.

Critical strengths
  1. The paper addresses a practical annotation problem without requiring gold labels during execution.
  2. It combines reliability estimation with cost-aware query selection.
  3. It reports consistent gains across five datasets, supported by ablations and robustness studies.
Critical weaknesses
  1. Although the paper focuses on LLMs, the method is restricted to fixed-label classification, including multiple-choice QA. Its core components (confusion-matrix estimation, dependent repeated annotations, and cost-aware acquisition) also apply to general noisy annotator pools. The authors should clarify what is specifically enabled by or designed for LLM annotators and scope their claims accordingly. The effectiveness of their method for open-ended generation tasks remains largely unexplored.

  2. CALE accounts for dependence between repeated answers from the same model, but does not explicitly model shared errors across different models. LLMs may have overlapping training data or similar weaknesses, so agreement of their answers does not always mean correctness. Testing pools with strongly correlated errors would show how well CALE handles this situation.

  3. The paper does not test sensitivity to initialization or the diagonal-biased reliability prior. Since CALE learns reliability without gold labels, these choices could affect which models it trusts and increase incorrect consensus. Testing different prior strengths and initializations shows the robustness of the method.

  4. Table 4 compares all aggregators using responses selected by CALE’s acquisition policy. Although this gives them the same evidence, the selected responses may particularly be in favor of CALE’s own aggregator. Testing the aggregators on responses collected by other policies, such as random selection and competing acquisition methods, would show whether its advantage holds more generally.

  5. The related-work discussion is a bit short and could include CrowdLayer (Rodrigues & Pereira, 2018), Tanno et al. (2019), and REALM (Ghiasvand et al., 2026), which also learn annotator reliability from noisy labels, while clarifying how CALE differs in its budgeted acquisition setting.

Questions for authors
  1. How sensitive is CALE to the reliability prior and initialization?
  2. How does it perform when several LLMs share the same systematic errors?
  3. Does CALE’s aggregator still outperform alternatives on responses collected by other acquisition policies?
  4. What is the total cost when embeddings and query-selection overhead are included?
2 comments

We sincerely thank the reviewer for the careful and in-depth evaluation. Below we provide detailed point-by-point responses.


Weakness1: Although the paper focuses on LLMs, the method is restricted to fixed-label classification, including multiple-choice QA. Its core components (confusion-matrix estimation, dependent repeated annotations, and cost-aware acquisition) also apply to general noisy annotator pools. The authors should clarify what is specifically enabled by or designed for LLM annotators and scope their claims accordingly. The effectiveness of their method for open-ended generation tasks remains largely unexplored.

  1. CALE's modeling and budget-allocation principles are not limited to LLM annotators.

    • Human annotation and human–LLM collaboration: Both settings involve heterogeneous annotators with different reliability and cost, so the same principles could guide label aggregation and query allocation under a shared budget.
  2. Classification is a controlled and practically important starting point.

    • Controlled evaluation: Since budgeted collaboration among heterogeneous LLM annotators remains underexplored, we begin with classification, including multiple-choice QA. This allows clear evaluation of annotation accuracy, model reliability, and budget efficiency while isolating repeated-response dependence from the complexities of free-form outputs.
    • Practical relevance: Classification annotation supports applications such as content moderation, intent detection, sentiment analysis, and document triage.
    • Open-ended direction: Our evaluation is limited to fixed-label tasks.

We thank the reviewer for this insightful comment, which highlights an important direction beyond the present evaluation.


Weakness2: CALE accounts for dependence between repeated answers from the same model, but does not explicitly model shared errors across different models. LLMs may have overlapping training data or similar weaknesses, so agreement of their answers does not always mean correctness. Testing pools with strongly correlated errors would show how well CALE handles this situation.

Same as Q2: How does it perform when several LLMs share the same systematic errors?

We appreciate this important observation. We examine it through a shared-confusion extension of CALE and a stress test of the original CALE with deliberately correlated responses, both using the paper's original datasets and evaluation protocol. AUBC (%) is reported as mean ±\pm sample standard deviation.

Experiment 1: Shared-error modeling. We extended CALE with an instance-level latent state HiH_i shared by all models:

Hi∣Yi=y,Ki=k∼Cat⁡(Sk,y,:).H_i \mid Y_i=y,K_i=k \sim \operatorname{Cat}(S_{k,y,:}).

ϕig∣Hi=h,Ki=k∼Dir⁡(κTg,k,h,:).\phi_{ig}\mid H_i=h,K_i=k \sim \operatorname{Dir}(\kappa T_{g,k,h,:}).

Aigv∣ϕig∼Cat⁡(ϕig).A_{igv}\mid\phi_{ig} \sim \operatorname{Cat}(\phi_{ig}).

Here, SkS_k is the shared confusion matrix and Tg,kT_{g,k} is the model-specific response matrix, giving the marginal confusion Ψg,k=SkTg,k\Psi_{g,k}=S_kT_{g,k}. Marginalizing HiH_i induces cross-model dependence, while the shared ϕig\phi_{ig} retains dependence between repeated responses from the same model.

MethodAG NewsMedNLIRTEMMLUDBpedia-14Avg.
Shared-base78.75 ± 0.2074.71 ± 0.4485.45 ± 0.7271.67 ± 0.4484.96 ± 0.1279.11 ± 0.16
CALE78.83 ± 0.2474.48 ± 0.2686.02 ± 0.6571.30 ± 0.2284.90 ± 0.1279.11 ± 0.14

Explicitly modeling systematic errors shared across models does not provide a clear AUBC advantage on these datasets, while adding one C×CC\times C matrix per regime, or KC(C−1)K C(C-1) additional free parameters, thereby increasing model complexity.

Experiment 2: Correlated-error stress test. We introduce ϵ\epsilon as the probability of fully copying one model's complete three-response label sequence to another model for the same item. A larger ϵ\epsilon therefore creates stronger repetition and cross-model dependence. We evaluate the setting ϵ=0.6\epsilon=0.6. The corresponding AUBC (%) results are:

MethodAG NewsMedNLIRTEMMLUDBpedia-14Avg.
DLTA75.98 ±\pm 0.1173.53 ±\pm 0.1284.22 ±\pm 0.5770.56 ±\pm 0.9885.19 ±\pm 0.1677.90 ±\pm 0.28
LMCrowd75.77 ±\pm 0.1873.94 ±\pm 0.3184.65 ±\pm 0.2169.65 ±\pm 0.5085.57 ±\pm 0.1477.92 ±\pm 0.09
CALE76.57 ±\pm 0.2373.78 ±\pm 0.2684.32 ±\pm 0.1270.02 ±\pm 0.8285.36 ±\pm 0.1378.01 ±\pm 0.16

CALE achieves the highest average AUBC, outperforming both DLTA and LMCrowd in this setting. This example shows that CALE can maintain a performance advantage when responses across models become substantially repetitive.

Weakness3: The paper does not test sensitivity to initialization or the diagonal-biased reliability prior. Since CALE learns reliability without gold labels, these choices could affect which models it trusts and increase incorrect consensus. Testing different prior strengths and initializations shows the robustness of the method.

Same as Q1: How sensitive is CALE to the reliability prior and initialization?

We thank the reviewer for suggesting these robustness checks. We varied one factor at a time across three axes:

  • Prior strength: Scale the default diagonal/off-diagonal pseudocounts (3.0, 1.1) by s=0.25 or 4, preserving the prior mean.
  • Diagonal bias: Set the diagonal-to-off-diagonal ratio to r=1.5 or 5, preserving the total pseudocount mass per row.
  • Initialization: Replace the default prior-mean initialization with uniform initialization, keeping the prior unchanged.
ConfigurationFinal Acc. Avg. (%) (mean ±\pm std)AUBC Avg. (%) (mean ±\pm std)
Prior strength s=0.2588.40 ±\pm 0.2478.60 ±\pm 0.30
Prior strength s=488.34 ±\pm 0.1878.90 ±\pm 0.19
Diagonal ratio r=1.588.23 ±\pm 0.2278.93 ±\pm 0.13
Diagonal ratio r=588.25 ±\pm 0.1779.14 ±\pm 0.15
Uniform initialization88.35 ±\pm 0.2879.07 ±\pm 0.14
Original88.43 ±\pm 0.1979.11 ±\pm 0.14

Across tested configurations and datasets, mean changes relative to the original fall within [-0.71, +0.68] percentage points for Final Accuracy and [-1.03, +0.26] points for AUBC. These results indicate limited sensitivity within the tested settings.


Weakness4: Table 4 compares all aggregators using responses selected by CALE’s acquisition policy. Although this gives them the same evidence, the selected responses may particularly be in favor of CALE’s own aggregator. Testing the aggregators on responses collected by other policies, such as random selection and competing acquisition methods, would show whether its advantage holds more generally.

Same as Q3: Does CALE’s aggregator still outperform alternatives on responses collected by other acquisition policies?

We thank the reviewer for suggesting this additional control. Random acquisition is independent of aggregator estimates and provides all aggregators with the same responses at B=4NB=4N; results are mean ±\pm standard deviation over five datasets and ten seeds per dataset:

AggregatorFinal Accuracy (%)AUBC (%)
MV67.71 ± 6.0351.82 ± 8.35
DS68.74 ± 5.5252.24 ± 8.21
GLAD68.69 ± 5.5452.20 ± 8.20
CALE68.70 ± 5.4852.22 ± 8.17

Acquisition determines the evidence available to aggregation, while aggregation's label and reliability estimates guide subsequent queries. New responses update both, so changing acquisition can change aggregation performance. Removing this feedback with Random acquisition lowers the performance of all methods relative to Table 4, but CALE remains comparable to the baselines.


Weakness5: The related-work discussion is a bit short and could include CrowdLayer (Rodrigues & Pereira, 2018), Tanno et al. (2019), and REALM (Ghiasvand et al., 2026), which also learn annotator reliability from noisy labels, while clarifying how CALE differs in its budgeted acquisition setting.

We thank the reviewer for recommending these related studies. We have added CrowdLayer, Tanno et al. (2019), and REALM to Section 5, “Related Work,” under “Crowdsourcing Aggregation”. Revised text and references are shown in blue.


Q4: What is the total cost when embeddings and query-selection overhead are included?

  1. Embedding overhead. We thank the reviewer for prompting us to make this accounting explicit. Embedding is a one-time preprocessing step per dataset. We measured embedding-request times using local nomic-embed-text with batch size 64:
DatasetAG NewsMedNLIRTEMMLUDBpedia-14Avg.
Embedding time (s)14.6214.2313.4514.0114.7214.21
  1. Acquisition-and-evaluation runtime. The table reports mean wall-clock time per seed over ten seeds for the acquisition-and-evaluation workflow, including query selection, periodic EM updates, evaluation, and aggregation comparisons using cached annotations. The computational setup is described in Appendix C.2. These timings exclude annotation-model inference; monetary annotation cost is determined by the model-specific query costs cgc_g
DatasetAG NewsMedNLIRTEMMLUDBpedia-14Avg.
Mean runtime(min)12.749.8115.888.4255.2020.41

We hope that our responses have adequately addressed the concerns raised. We respectfully ask if you would be comfortable reconsidering the rating in light of these efforts, and we remain happy to address any further questions.

Reviewer_Cg7ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on the setting of adaptive sequential sampling and label aggregation of heterogeneous LLM-generated annotations from a fixed pool of samples in order to produce a set of high-fidelity annotations. Labels, individual and joint annotator reliability are assumed unknown a priori. The proposed budget allocation and ensemble framework, CALE, additionally incorporates an objective based on cost of label acquisition from the model ensemble. CALE uses a latent Bayesian model to jointly reflect the data generating process underlying model labels on items and leverages the inferred model parameters to aggregate collected labels. The sequential sampling process similarly exploits the learned parameters in order to compute possible posteriors over the label domain, and chooses the subsequent instance that maximizes a joint improvement in label fidelity and reliability information, normalized by cost. The paper demonstrates the utility of their method on a wide range of tasks with heterogeneous LLM pools, particularly at low budgets.

Critical strengths
  • Good performance under the settings investigated, and substantial improvement over relevant baselines in several cases
  • Methodology is sound and well-reasoned
  • Thorough and reasonable presentation
  • Ablative experiments are excellent and very useful for understanding in what scenarios different components of their method improve performance
Critical weaknesses
  • Choice of budget issues (willing to raise my score if this is addressed):
    • Low budget setting is stated as the most useful or prominent setting for when CALE improves over other methods but according to Figure 3, this only seems to be the case in two datasets, and CALE loses accuracy in another.
    • The choice of B=N where N=number of objects in the entire dataset as the floor for the budget seems arbitrary and the comparable baseline (DTLA) and CALE seem to have largely converged by that point.
  • Methodology differentiation over DTLA + improvement over their method; seems fairly related to DTLA applied to LLM settings, extended largely to include the heterogeneity regime component (willing to raise my score if this is differentiated well in author response)
Questions for authors
  • Can the authors motivate why N (the number of objects in the dataset total, roughly 3000) is chosen as the floor for the budget? Maybe I’m misunderstanding but I feel like realistically, we would want to see the performance at much smaller budgets than N (since many of these datasets have thousands of items). What about the performance of the method on the order of ~100 items?
  • Is the latent regime intended to capture the heterogeneity across models, or model-error patterns? This was my interpretation, as this seems to be the primary component that handles heterogeneity.
  • There isn't much in the way of discussion about if the proposed Bayesian model fails to represent the true nature of the label data generating process. This is okay methodologically, but should be mentioned as a limitation probably

Minor notes to improve paper:

  • In the abstract, it’s unclear what N is when used in the context of the accuracy in the final few sentences? I think you should mention what this is or cut it and just have relative statements, this was confusing to read
  • Missing some possible additional citations of LLM annotator inference/evaluation (possibly irrelevant, and fair to justify if so): https://arxiv.org/pdf/2601.20251, https://arxiv.org/abs/2403.03208, other active sampling/label acquisition papers
3 comments

We thank the reviewer for the careful assessment and the concrete suggestions. We respond to the budget concerns in Weakness 1 and Q1 together, then address the remaining points individually.


Weakness1: Choice of budget issues (willing to raise my score if this is addressed)

Also Q1: Can the authors motivate why N (the number of objects in the dataset total, roughly 3000) is chosen as the floor for the budget? Maybe I’m misunderstanding but I feel like realistically, we would want to see the performance at much smaller budgets than N (since many of these datasets have thousands of items). What about the performance of the method on the order of ~100 items?

We thank the reviewer for identifying an ambiguity in Figure 3. The revised figure shows the full accuracy–cost trajectories from 00 to 4N4N, including budgets below NN. Since a small-model call costs one unit, NN is the cost of querying each of the NN items once if all calls use small models; it can be viewed as a stress scenario: a limiting budget that covers all problems at the lowest possible cost. Below NN, the curves for all models rise rapidly as the policies expand coverage. This coverage-limited range, including the reviewer's roughly 100-items.

Prior work on budgeted crowdsourcing frames practical allocation as a trade-off between covering new items and obtaining additional labels for uncertain ones (Chen et al., 2015; Ho et al., 2013). In our normalized-cost protocol, this trade-off becomes meaningful once initial coverage has been established: budgets at or above roughly 2N2N allow one cheap call per item while leaving resources for repeated labels on difficult items. We therefore regard 2N2N--4N4N as an illustrative finite-budget operating range, not as a universal standard. By contrast, when B<NB<N, the regime is coverage-limited: even if all queries use the cheapest models, the budget cannot cover every item once, so querying a previously untouched item can yield a larger marginal gain than repeatedly labeling an already covered item. Figure 3 shows that in the 2N2N--4N4N range, CALE maintains high and comparatively stable accuracy across all five datasets, whereas LMCrowd reaches an early peak and then declines on several datasets as the budget increases. Thus, NN is a stringent early-budget stress test, while the broader trajectory shows CALE's robustness across the illustrated coverage--refinement range rather than at a selected endpoint alone.

Reference:

Weakness2: Methodology differentiation over DTLA + improvement over their method; seems fairly related to DTLA applied to LLM settings, extended largely to include the heterogeneity regime component (willing to raise my score if this is differentiated well in author response).

We thank the reviewer for highlighting DLTA as a close predecessor. We agree that dynamically coupling label inference and acquisition is not new; our contribution is instead a different probabilistic model and acquisition objective for heterogeneous, repeatedly sampled LLM annotators.

  1. Regime-shared, instance-specific model reliability. DLTA models a worker's reliability and an item's easiness with scalar parameters, so its response model cannot represent which incorrect class a particular worker tends to produce on a particular type of item. CALE instead uses a class-conditional confusion matrix Ψg,k\boldsymbol\Psi_{g,k} for each model gg and latent instance regime kk. The feature-informed prior over ZiZ_i allows reliability patterns to be shared across similar instances, while Φi,g∣Zi=k∼Dir(κΨg,k)\boldsymbol\Phi_{i,g}\mid Z_i=k\sim\mathrm{Dir}(\kappa\boldsymbol\Psi_{g,k}) allows instance-specific departures. The regime therefore organizes model-error patterns across instances; model identity remains explicit in gg.

  2. Dependence among repeated calls. DLTA assumes labels are conditionally independent given the item label and worker/item parameters, and its acquisition procedure obtains an item's additional labels from different workers. Repeated stochastic calls to the same LLM on the same instance require a different observation model: in CALE, those responses share the latent row Φi,g,Yi\boldsymbol\Phi_{i,g,Y_i}. Integrating it out yields a Dirichlet--compound--multinomial likelihood and a posterior-predictive response distribution conditioned on earlier calls. Thus, another agreeing response need not count as an independent vote, and its expected marginal value can diminish.

  3. Additional model-reliability learning gain as an acquisition consideration. DLTA allocates labels using the expected gain in confidence for the queried item. CALE evaluates a costed instance--model query using both its expected reduction in that item's label-decision risk, Δlocal\Delta_{\mathrm{local}}, and Δparam\Delta_{\mathrm{param}}, a tractable surrogate for reducing uncertainty in shared model--regime confusion rows. The latter gives acquisition a reason to learn a model's error behavior for later decisions even when the current item's label is unlikely to change. The combined score is normalized by model-specific cost cgc_g; Δparam\Delta_{\mathrm{param}} is an exploration surrogate, not exact long-horizon information gain.

Our methodological advance over DLTA is to couple regime-to-instance reliability sharing and dependence-aware repeated-call inference with a cost-aware acquisition rule that values both current labels and future reliability learning. These designs respond to the distinctive characteristics of LLM annotation pools—heterogeneous error patterns, model-specific costs, and dependence among repeated calls to the same model—rather than merely substituting LLMs for crowd workers.

Q2: Is the latent regime intended to capture the heterogeneity across models, or model-error patterns? This was my interpretation, as this seems to be the primary component that handles heterogeneity.

Thank you for the question. The latent regime is intended to capture instance-dependent model-error patterns, whereas cross-model heterogeneity is represented by model-specific confusion matrices. In other words, the regime is an instance-side sharing variable; it is not a substitute for the model index.

To illustrate the model-specific component, below are fitted AG News confusion matrices at the final budget (seed 1). Rows are hypothesized true classes and columns are model outputs, in the order [World, Sports, Business, Sci/Tech]. The probabilities are fitted without access to gold labels.

DeepSeek-R1

Hypothesized true class / outputWorldSportsBusinessSci/Tech
World0.83380.03020.07570.0603
Sports0.00820.97430.01310.0044
Business0.00620.00320.98330.0073
Sci/Tech0.03440.00560.35860.6014

Phi-4

Hypothesized true class / outputWorldSportsBusinessSci/Tech
World0.98110.00500.01000.0039
Sports0.17860.80380.01170.0060
Business0.39610.00540.58630.0122
Sci/Tech0.46950.00980.18080.3399

Qwen2.5

Hypothesized true class / outputWorldSportsBusinessSci/Tech
World0.88560.01910.04330.0520
Sports0.01470.96280.01070.0118
Business0.02160.00640.70120.2708
Sci/Tech0.00710.01260.01970.9606

Gemma4

Hypothesized true class / outputWorldSportsBusinessSci/Tech
World0.61610.06070.14360.1797
Sports0.02530.92440.02510.0252
Business0.03140.02640.71610.2260
Sci/Tech0.03800.03040.38530.5462

The fitted matrices distinguish model-specific error directions. The off-diagonal mass in the fitted matrices is concentrated on different class transitions. DeepSeek-R1 and Gemma4 show prominent Sci/Tech-to-Business confusion (35.86% and 38.53%, respectively), whereas Phi-4 tends to predict World for both Business and Sci/Tech (39.61% and 46.95%, respectively). Qwen2.5 shows a different, asymmetric pattern: Business-to-Sci/Tech confusion is substantial (27.08%), while the reverse transition is rare (1.97%). Thus, model heterogeneity concerns not only how likely an error is, but also which incorrect label is produced. These model-specific patterns are represented by the index gg in Ψg,k\Psi_{g,k}.


Q3: There isn't much in the way of discussion about if the proposed Bayesian model fails to represent the true nature of the label data generating process. This is okay methodologically, but should be mentioned as a limitation probably.

Thank you for raising this limitation. As with generative Bayesian models such as LDA, CALE provides a structured approximation to the response-generating process, not a guarantee that the assumed process exactly matches real LLM behavior. In particular, it models dependence among repeated calls to the same model but assumes no residual dependence across models after conditioning on the latent regime; shared systematic errors can therefore make agreement appear stronger than it is. Misspecification may lead to overconfident label posteriors and misestimated reliability, which can also affect subsequent acquisition decisions. We acknowledge this limitation and will state it explicitly in the limitations discussion; the cross-model conditional-independence assumption and its potential effect on confidence are already noted in the appendix.


Minor notes

Thank you for helping us clarify the notation and improve the coverage of related work. We have revised the following two parts of the PDF, with the changes marked in blue:

  1. Abstract (p. 1): Definition of NN. We now introduce NN as the number of instances in the fixed annotation pool.

  2. Section 5, Related Work (p. 9): Additional references. Under “Cost-Aware Acquisition,” we added both suggested papers and briefly distinguish their focus on statistical inference and LLM benchmark evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.