- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ICC divides oracle calls between collecting ICL demonstrations and escalating uncertain predictions. The paper proposes harvest strategies and budget-allocation rules, with theoretical analyses and experiments on four binary classification datasets.
Critical strengths
- Reusing oracle labels as demonstrations is practically useful. At a 5% counted oracle budget, ICC improves Overruling accuracy from 0.875 to 0.966 and ToxiGen from 0.783 to 0.877 over FixedCascade (Table 1).
Critical weaknesses
- Theorem 3's proof conflicts with Definition 5. Coverage phi_k already denotes the probability of harvesting an ambiguous item, but the proof uses phi_k times delta. This double-counts ambiguity and produces the stated inverse-delta-squared interpretation for random harvesting, unlike Theorem 1. The strategy guarantee is not established by this proof.
- The smallest-budget claims rely on substantial supervision outside B: the main setup supplies 50 labeled calibration items, while B is only 10 for a 1,000-item pool at 1%. Appendix C.5 studies calibration sensitivity at 5%, leaving the total-label savings at the tightest budgets unresolved.
Questions for authors
- Can Theorem 3 be rederived consistently with phi_k's definition and the actual deterministic or without-replacement harvest procedures?
- How do gains change when calibration and pilot labels count toward the total budget, and baselines may reuse those labels as demonstrations?
2 comments
Reminder about uploaded revision
As the discussion period closes, we wanted to leave a brief final note. We have uploaded a revised PDF with all changes marked in blue, and our rebuttal above summarizes how it addresses your feedback.
In particular, we corrected Theorem 3 and made the label accounting explicit.
We would be grateful if you would take the revised version into account when finalizing your evaluation, and we hope you will consider raising your score. Thank you again for your time and for improving the paper.
Rebuttal
Thank you for your careful review and for pointing out the conflict between Theorem 3 and Definition 5. We have uploaded a revised PDF in which all changes are marked in blue.
W1 and Q1: Theorem 3. You are correct. Since
φ_kis already the per-item probability of harvesting an ambiguous item (Definition 5), the success probability isφ_k, notφ_k·δ. We re-derived the bound in Appendix A:B_h ≥ (k* + L + √(L² + 2k*L)) / φ_kwithL = log(1/α), where the numerator comes from our corrected Theorem 1 (multiplicative Chernoff bound). For random harvest, this givesk*/δ, consistent with Theorem 1 and no longerk*/δ². For uncertainty harvest, it gives aboutk*. The ratioφ_unc/δis unchanged, so the qualitative conclusion stands. On the harvest procedures, we note that random harvest satisfies (A1) exactly, and sampling without replacement gives a hypergeometric count that obeys the same Chernoff bound. Uncertainty and diversity harvest are deterministic given the confidence scores, so there the bound is the budget required under an independent-sampling model with success probabilityφ_k. We now state this in the theorem discussion and in the proof. No allocation rule depends on Theorem 3, and no reported result changes.W2 and Q2: Labels outside B. We now state explicitly in Section 2 and in the Evaluation paragraph of Section 4 that the 50 calibration labels are outside
B, are used by all calibrated baselines, and are reused by the allocation pilots, so no additional labels are required. We report the total asB+ 50 and give the example you raise: a nominal 1% budget on a 1,000-item pool is 10 pool labels plus the 50 calibration labels, so we no longer present it as a 1% total annotation requirement. Appendix C.5 varies the calibration set size. A baseline that reuses the 50 calibration labels as ICL demonstrations is a natural extension that we will investigate and report in the final version.We believe the revision resolves the soundness concerns, as the error is corrected without changing any empirical result. We would therefore be grateful if you would consider raising your score. Thank you again for your valuable review.