- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CALE combines hierarchical label aggregation with cost-aware acquisition to annotate a fixed dataset using heterogeneous LLMs under a shared budget. Building on confusion-matrix aggregation, it models regime-specific reliability and dependence among repeated calls to the same model on the same example. Query selection balances expected label improvement and a surrogate for learning reliability against query cost. The experiments cover five text-classification and multiple-choice benchmarks.
Critical strengths
- Promising empirical performance. The method achieves the highest reported area under the accuracy–cost curve on all five benchmarks, supporting its usefulness for budget-constrained annotation (Table 1, p. 7).
- Relevant modeling choices. The framework incorporates regime-specific reliability, repeated-call dependence, and heterogeneous query costs into a coupled acquisition and aggregation procedure.
- Component analyses. The hierarchy, acquisition, and aggregation ablations help assess the contributions of the method’s components (Tables 2–4, p. 8), complemented by sensitivity analyses in Appendix G.
Critical weaknesses
-
The incremental benefit of the proposed aggregation needs stronger evidence.
With the acquisition history held fixed, Table 4 reports average final accuracy of 88.43% for CALE, compared with 87.92% for majority vote and 87.82% for Dawid–Skene. These improvements are modest, and the table does not report significance tests or confidence intervals for the paired differences. The corresponding average AUBC gains are also small: 0.20 and 0.15 points, respectively (Table 10, p. 18).This does not establish that the hierarchical aggregator is unnecessary, since it also guides acquisition. However, stronger evidence is needed to distinguish its contribution to final label aggregation from its contribution to query selection, and to justify the added complexity.
-
Robustness to correlated errors across models remains unclear.
CALE explicitly models dependence among repeated calls to the same model, while regimes can capture some shared difficulty patterns. However, residual cross-model dependence remains unmodeled, as acknowledged in Appendix B (p. 14). Models sharing systematic mistakes could produce misleading agreement and overconfident label estimates.This limitation is also shared by standard Dawid–Skene, so my concern is whether CALE retains its empirical advantage over the baselines under strongly correlated model errors. A controlled analysis would help establish the scope of its dependence-aware formulation.
Questions for authors
- Are the aggregation improvements in Table 4 statistically significant under a paired analysis across the shared acquisition histories? Can you clarify how much the hierarchical aggregator contributes through query selection versus final label aggregation?
- How does CALE perform relative to the baselines when different models have strongly correlated errors, such as repeatedly making the same mistakes?
2 comments
Thank you for the detailed rebuttal and additional experiments. The paired statistical analysis adequately addresses my concern about the significance of aggregation improvements, although the effect sizes remain modest. The correlated-error experiments also provide useful evidence of robustness, despite relatively small gains over baselines.
Overall, the new evidence addresses my main concerns and strengthens the paper. I am updating my rating from 2 (Weak Reject) to 3 (Weak Accept).
We sincerely thank the reviewer for the detailed and thoughtful evaluation. Below we provide detailed point-by-point responses.
Weakness1: The incremental benefit of the proposed aggregation needs stronger evidence.
We thank the reviewer for highlighting the need for paired statistical evidence and a clearer explanation of the aggregator's contribution.
Final aggregation on shared acquisition histories. We compared CALE with alternative aggregators on the same 50 dataset--seed acquisition histories. The paired differences are:
The confidence intervals use paired bootstrap resampling within each dataset, and the p-values use a two-sided paired Wilcoxon test with Holm correction. Thus, although the average gains are modest, the final aggregation improvement is statistically supported when the evidence is held fixed.
Interaction with query selection. Because aggregation and acquisition form a closed loop in the full method, the total improvement cannot be uniquely or additively decomposed into an “aggregation percentage” and an “acquisition percentage.” The fixed-history analysis above isolates the conditional benefit of final aggregation, while the end-to-end improvement reflects their joint interaction.
Weakness2: Robustness to correlated errors across models remains unclear.
We appreciate this important observation. We examine it through a shared-confusion extension of CALE and a stress test of the original CALE with deliberately correlated responses, both using the paper's original datasets and evaluation protocol. AUBC (%) is reported as mean ± sample standard deviation.
Experiment 1: Shared-error modeling. We extended CALE with an instance-level latent state Hi shared by all models:
Hi∣Yi=y,Ki=k∼Cat(Sk,y,:).
ϕig∣Hi=h,Ki=k∼Dir(κTg,k,h,:).
Aigv∣ϕig∼Cat(ϕig).
Here, Sk is the shared confusion matrix and Tg,k is the model-specific response matrix, giving the marginal confusion Ψg,k=SkTg,k. Marginalizing Hi induces cross-model dependence, while the shared ϕig retains dependence between repeated responses from the same model.
Explicitly modeling systematic errors shared across models does not provide a clear AUBC advantage on these datasets, while adding one C×C matrix per regime, or KC(C−1) additional free parameters, thereby increasing model complexity.
Experiment 2: Correlated-error stress test. We introduce ϵ as the probability of fully copying one model's complete three-response label sequence to another model for the same item. A larger ϵ creates stronger repetition and cross-model dependence. We study ϵ=0.6. AUBC (%) results:
CALE achieves the highest average AUBC, outperforming both DLTA and LMCrowd in this setting. This example shows that CALE can maintain a performance advantage when responses across models become substantially repetitive.
We hope that our responses have adequately addressed the concerns raised. We respectfully ask if you would be comfortable reconsidering the rating in light of these efforts, and we remain happy to address any further questions.