acceptodds
Under review as a conference paper at ICLR 2027

In-Context Cascade: Few-Shot Adaptation for Budget-Constrained Labeling at Scale

Abstract

Labeling large pools of unlabeled text items with respect to a natural language predicate is a common bottleneck in data annotation, content moderation, and enterprise AI pipelines. While large language models can automate this task, exhaustive labeling by a costly oracle such as a frontier model or a human annotator is infeasible for pools of thousands to millions of items. We propose the In-Context Cascade (ICC), a budget-constrained pool labeling framework that combines in-context learning (ICL) adaptation of a cheap model with targeted oracle escalation. ICC allocates a small harvest budget from total budget to oracle-label items near the decision boundary, uses these as ICL demonstrations to adapt the cheap model across the remaining pool, and reserves an escalation budget to resolve residual uncertainty. We provide theoretical guarantees on coverage bounds, harvest strategy comparison, optimal budget allocation, and escalation thresholds, leveraging these for the algorithm. Experiments on four datasets with oracle budget from to of pool size show that ICC consistently outperforms uncertainty-based cascade baselines, with the largest gains often at the tightest budgets where ICL adaptation is most valuable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cascade, budget-constrained labeling, in-context learning, active learning, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TDYHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ICC divides oracle calls between collecting ICL demonstrations and escalating uncertain predictions. The paper proposes harvest strategies and budget-allocation rules, with theoretical analyses and experiments on four binary classification datasets.

Critical strengths
  • Reusing oracle labels as demonstrations is practically useful. At a 5% counted oracle budget, ICC improves Overruling accuracy from 0.875 to 0.966 and ToxiGen from 0.783 to 0.877 over FixedCascade (Table 1).
Critical weaknesses
  1. Theorem 3's proof conflicts with Definition 5. Coverage phi_k already denotes the probability of harvesting an ambiguous item, but the proof uses phi_k times delta. This double-counts ambiguity and produces the stated inverse-delta-squared interpretation for random harvesting, unlike Theorem 1. The strategy guarantee is not established by this proof.
  2. The smallest-budget claims rely on substantial supervision outside B: the main setup supplies 50 labeled calibration items, while B is only 10 for a 1,000-item pool at 1%. Appendix C.5 studies calibration sensitivity at 5%, leaving the total-label savings at the tightest budgets unresolved.
Questions for authors
  1. Can Theorem 3 be rederived consistently with phi_k's definition and the actual deterministic or without-replacement harvest procedures?
  2. How do gains change when calibration and pilot labels count toward the total budget, and baselines may reuse those labels as demonstrations?
2 comments

Thank you for your careful review and for pointing out the conflict between Theorem 3 and Definition 5. We have uploaded a revised PDF in which all changes are marked in blue.

W1 and Q1: Theorem 3. You are correct. Since φ_k is already the per-item probability of harvesting an ambiguous item (Definition 5), the success probability is φ_k, not φ_k·δ. We re-derived the bound in Appendix A: B_h ≥ (k* + L + √(L² + 2k*L)) / φ_k with L = log(1/α), where the numerator comes from our corrected Theorem 1 (multiplicative Chernoff bound). For random harvest, this gives k*/δ, consistent with Theorem 1 and no longer k*/δ². For uncertainty harvest, it gives about k*. The ratio φ_unc/δ is unchanged, so the qualitative conclusion stands. On the harvest procedures, we note that random harvest satisfies (A1) exactly, and sampling without replacement gives a hypergeometric count that obeys the same Chernoff bound. Uncertainty and diversity harvest are deterministic given the confidence scores, so there the bound is the budget required under an independent-sampling model with success probability φ_k. We now state this in the theorem discussion and in the proof. No allocation rule depends on Theorem 3, and no reported result changes.

W2 and Q2: Labels outside B. We now state explicitly in Section 2 and in the Evaluation paragraph of Section 4 that the 50 calibration labels are outside B, are used by all calibrated baselines, and are reused by the allocation pilots, so no additional labels are required. We report the total as B + 50 and give the example you raise: a nominal 1% budget on a 1,000-item pool is 10 pool labels plus the 50 calibration labels, so we no longer present it as a 1% total annotation requirement. Appendix C.5 varies the calibration set size. A baseline that reuses the 50 calibration labels as ICL demonstrations is a natural extension that we will investigate and report in the final version.

We believe the revision resolves the soundness concerns, as the error is corrected without changing any empirical result. We would therefore be grateful if you would consider raising your score. Thank you again for your valuable review.

AuthorsOpenReview ↗

Reminder about uploaded revision

As the discussion period closes, we wanted to leave a brief final note. We have uploaded a revised PDF with all changes marked in blue, and our rebuttal above summarizes how it addresses your feedback.

In particular, we corrected Theorem 3 and made the label accounting explicit.

We would be grateful if you would take the revised version into account when finalizing your evaluation, and we hope you will consider raising your score. Thank you again for your time and for improving the paper.

Reviewer_AqKBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes In-Context Cascade (ICC) for labeling a fixed pool of text under a limited oracle budget. It divides the budget between obtaining labeled examples for in-context adaptation of a cheaper model and directly resolving uncertain predictions afterward. The authors introduce several selection and allocation strategies, supported by four theoretical results. Experiments cover four binary classification datasets, with additional model comparisons on Overruling. The central idea is to use purchased labels both as final annotations and as demonstrations that improve predictions on other items.

Critical strengths
  1. I find the practical motivation compelling. Reusing oracle-labeled examples to improve the cheap model is a sensible way to obtain more value from a limited annotation budget, and the method does not require updating model parameters.

  2. The empirical improvements over FixedCascade are encouraging. At a 5% oracle budget, the ambiguous-region allocation improves accuracy from 0.875 to 0.966 on Overruling and from 0.783 to 0.877 on ToxiGen. I also appreciate the additional comparisons across four models in Appendix C.1 and the experiments separating harvesting, demonstration selection, and escalation.

Critical weaknesses
  1. The theoretical guarantees are not valid as stated. In Theorem 1, take k^{*}=10, δ=0.1, and α=0.5. The stated condition is satisfied by Bₕ=70, but under the theorem’s own binomial assumption,
Pr⁡[Binomial⁡(70,0.1)≥10]≈0.1586,\Pr[\operatorname{Binomial}(70,0.1)\geq10]\approx0.1586,
 which is below the required probability of 0.5. The final step of the
 proof does not follow from the preceding sufficient conditions.
 Separately, Definition 5 defines ϕₖ as the expected fraction of harvested
 items in the ambiguous region, but the proof of Theorem 3 treats the
 success probability as ϕₖδ. For random harvesting, this gives δ², whereas
 the actual probability is δ. This double counting undermines the stated
 harvest-budget scaling. Since these results motivate the allocation
 methods, correcting them and identifying the consequences for the
 algorithm are necessary.

2. The practical cost and scaling claims are stronger than the evaluation establishes. The main experiments assume a separate set of 50 labeled calibration items. For a pool of 1,000 items, the nominal 1% oracle budget therefore means 10 pool-labeling calls in addition to those 50 labels. This is consistent with the stated problem setup, but it does not establish a 1% total annotation requirement when starting without labeled data. The reported curves also exclude the cost of the initial cheap-model pass and the longer demonstration-based prompts. With evaluated pools of only 1,000–2,400 items, the paper provides limited evidence for the broader claim of cost-effective labeling at scale.

Questions for authors
  1. How would you correct Theorems 1 and 3, including the counterexample and the extra factor of δ above? Which allocation rules or claimed guarantees change after these corrections?

  2. Can you report the comparison with calibration labels included in the total annotation budget, and provide an accounting of cheap-model input tokens or monetary cost? Please also clarify whether the allocation pilots reuse the existing calibration labels or require additional labeled examples.

2 comments

Thank you for your careful review, and in particular for the explicit counterexample. We have uploaded a revised PDF in which all changes are marked in blue.

W1 and Q1: Theorems 1 and 3. You are right on both points, and the corrections are in the revision (Section 3.2 and Appendix A).

  • Theorem 1. The last step of the proof did not follow from the preceding sufficient conditions. We re-derived the bound with a multiplicative Chernoff bound: with L = log(1/α), harvesting B_h ≥ (k* + L + √(L² + 2k*L)) / δ items ensures at least k* ambiguous demonstrations with probability at least 1 − α. For your example (k* = 10, δ = 0.1, α = 0.5) this gives B_h = 145, with exact binomial probability 0.923 ≥ 0.5. The scaling in 1/δ is unchanged.
  • Theorem 3. The per-item success probability is φ_k (Definition 5), not φ_k·δ. For random harvest the budget scales as k*/δ rather than k*/δ², and for uncertainty harvest it is about k*. The ratio φ_unc/δ is unchanged. We also state that (A1) is exact for random harvest and an idealization for the deterministic harvests.
  • Key changes. The target-demonstrations rule β = min(1, k* log(1/α) / (δ̂ B)) is now described as a simplified heuristic. At the defaults used in all experiments (k* = 8, α = 0.1), it satisfies the corrected bound (coefficient 18.4 vs. 16.8 required; exact probability above 0.998 for the δ values we checked), so no reported result changes. The ambiguous-region rule (β = δ̂) and the fitted rules do not depend on Theorems 1 or 3, and no rule depends on Theorem 3.

W2 and Q2: Cost and scaling claims. We now state explicitly in Section 2 and in the Evaluation paragraph of Section 4 that the 50 calibration labels are outside B, are used by all calibrated baselines, and are reused by the allocation pilots (no additional labeled examples are required). We also report the total as B + 50, with the worked example you describe: a nominal 1% budget on a 1,000-item pool is 10 pool labels plus the 50 calibration labels, and we no longer present it as a 1% total annotation requirement. Appendix C.5 varies the calibration set size. We added an inference-cost paragraph, mentioning that all methods share one cached zero-shot pass, ICC adds one ICL pass with prompts capped at k_max demonstrations, and cost is linear in N. ICC's cheap-model cost is higher than that for FixedCascade, and we mention this explicitly. On the subject of scale, we revised the text to remove any implication of large-pool evaluation; our pools are limited by the need for ground truth on every item, and Theorems 1–4 do not depend on N.

We believe the revision resolves the soundness concerns, since both errors are corrected without altering any empirical result. We would therefore be grateful if you would consider raising your score. Thank you again for your time and for feedback that helped us improve the paper.

AuthorsOpenReview ↗

Reminder about uploaded revision

As the discussion period closes, we wanted to leave a brief final note. We have uploaded a revised PDF with all changes marked in blue, and our rebuttal above summarizes how it addresses your feedback.

In particular, we corrected Theorems 1 and 3, and made the label and cost accounting explicit.

We would be grateful if you would take the revised version into account when finalizing your evaluation, and we hope you will consider raising your score. Thank you again for your time and for improving the paper.

Reviewer_9XQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes In-Context Cascade (ICC), a framework for budget-constrained large-scale text annotation. The central idea is to exploit expensive oracle annotations for two purposes: adapting a cheaper language model through in-context learning (ICL) and directly correcting uncertain predictions through oracle escalation. The proposed framework consists of five stages: preprocessing, budget allocation, demonstration harvesting, ICL-based prediction, and oracle escalation. The authors provide theoretical analyses of demonstration coverage, budget allocation, harvesting strategies, and escalation thresholds. Experiments on four binary classification datasets show that ICC generally outperforms conventional cascade-based annotation methods, particularly under limited oracle budgets.

Critical strengths
  1. The paper addresses a practically relevant problem. Large-scale data annotation with limited access to expensive annotators is an important problem. Improving annotation quality while controlling the number of expensive oracle calls has clear practical value.
  2. The proposed framework is intuitive and reasonably well organized. The idea of using oracle annotations both as ICL demonstrations and as direct corrections is sensible. The five-stage pipeline provides a clear framework for combining task adaptation with selective escalation.
  3. The experimental results demonstrate promising improvements. The proposed method generally achieves higher annotation accuracy than the evaluated cascade baselines across four datasets and several oracle budget levels. The improvements are particularly noticeable when the oracle budget is limited.
Critical weaknesses
  1. The theoretical analysis contains questionable assumptions and potentially incorrect derivations. The theoretical results are presented as an important contribution, but several arguments require substantial clarification. In particular, Theorem 3 appears to incorrectly characterize the probability that a harvested instance belongs to the ambiguous region. Since the ambiguous coverage \phi_k is already defined as the proportion of harvested instances belonging to this region, multiplying it by the overall ambiguous-region proportion \delta appears to double-count the same probability.
  2. The proposed budget formulation does not adequately account for all annotation and inference costs. The framework explicitly constrains the number of expensive oracle calls, but the preprocessing stage requires a labeled calibration set, and some allocation strategies additionally require pilot experiments. The paper does not clearly explain how these costs are incorporated into the total oracle budget. Furthermore, the cost of repeated ICL inference is not explicitly considered in the optimization objective. Consequently, the claimed cost efficiency may not fully reflect the actual deployment cost.
  3. The experimental evaluation is relatively limited for the claimed large-scale annotation setting. The experiments involve only four binary classification datasets, with pool sizes of approximately 1000–2400 instances. These experiments provide preliminary evidence of effectiveness but do not sufficiently demonstrate scalability to the much larger annotation pools emphasized in the introduction. Furthermore, the experiments simulate a perfect oracle using ground-truth labels. The effectiveness of the framework under realistic conditions involving imperfect human or LLM annotators remains unclear.
  4. The novelty of the individual components is somewhat limited. The proposed framework combines several established techniques, including uncertainty sampling, in-context learning, confidence calibration, and selective prediction. While their integration is potentially useful, the paper should more carefully distinguish its methodological contributions from existing work on active learning, demonstration selection, and LLM cascades. In particular, the empirical improvements appear to be largely driven by introducing ICL adaptation into the cascade. The additional benefits of the proposed theoretical allocation and harvesting strategies are less convincing.
Questions for authors

Please see the Weaknesses.

2 comments

Thank you for your careful reading and for the constructive feedback. We have uploaded a revised PDF in which all changes are marked in blue. Our responses to your questions are below, with locations in the revision.

W1: Theorem 3 double-counts δ. Thank you for catching this issue. Since φ_k is already the probability that a harvested item lies in the ambiguous region, the per-item success probability is φ_k, not φ_k·δ. We corrected the statement and proof of Theorem 3 (Appendix A). For random harvest, the required budget scales as k*/δ (not k*/δ²), and for uncertainty harvest it is about k*. The ratio of the random to the uncertainty budget, φ_unc/δ, and hence the qualitative conclusion, is unchanged. We also fixed an error in the final step of Theorem 1 (the bound is now derived with a multiplicative Chernoff bound) and noted which allocation rule is affected. At the defaults used in all experiments (k* = 8, α = 0.1) the rule still satisfies the corrected guarantee, so no reported result changes. We also clarify that assumption (A1) (independent harvest draws) holds exactly for random harvest, and that for the deterministic uncertainty and diversity harvests, the bound is the budget required under an independent-sampling model with success probability φ_k.

W2: Budget and cost accounting. We now state in Section 2 and in the Evaluation paragraph of Section 4 that the 50 calibration labels are outside B, are used by all calibrated baselines, and are reused for the allocation pilots, so no extra labels are needed. We also give the total label count, B + 50, with a worked example for a 1,000-item pool. We added an inference-cost paragraph at the end of Section 4.1, mentioning that all methods share one cached zero-shot pass, ICC adds one ICL pass with prompts capped at k_max demonstrations, and the cost is linear in N. We agree that ICC's cheap-model cost is higher than that for FixedCascade, but it is incurred on a low-cost model rather than through oracle calls.

W3: Scale and perfect oracle. Our pools are limited to 1,000–2,400 items because controlled evaluation needs ground-truth labels for every item. Theorems 1–4 do not depend on N: the harvest budget needed for k* ambiguous demonstrations is fixed in absolute terms, so the required fraction B_h/N falls as N grows. We revised the text to say this and to avoid implying we evaluated larger pools. For the oracle, we now state how oracle error would enter ICC – as an error floor on escalated labels and noise in demonstrations. Noisy-oracle evaluation is a useful direction for future work.

W4: Novelty and source of gains. To our knowledge, ICC is the first framework to use each oracle call both as a final label and as an ICL demonstration under one fixed budget, and to analyze how that budget should be split. Existing cascades keep the cheap model fixed and are the special case β = 0. The individual components are established, as we now say in the Introduction. The contribution is the allocation problem they create and its solution: coverage guarantees for harvesting (Theorems 1 and 3), the optimal split (Theorem 2), optimal escalation thresholds (Theorem 4), and allocation rules that need no ICL pilot calls. Related in-context distillation work (Sarukkai et al.; Wu et al.), now discussed in the Introduction, targets agentic tasks or online streams and does not study how to divide a fixed oracle budget. The revised Allocation Method Comparison paragraph also separates the sources of gain. Pure harvest improves on FixedCascade by 3–7 points (ICL adaptation). The ambiguous-region allocation adds 5.2 and 2.6 points over pure harvest on Overruling and ToxiGen and 0.7 on Headlines, and is 1.6 points lower on Movie Reviews, so both sources of gain are measured.

We hope the revision addresses your concerns. Given that the theoretical errors you identified are corrected without changing any empirical result, and that the accounting and scope are now explicit, we would be grateful if you would consider raising your score. Thank you again for your thoughtful review.

AuthorsOpenReview ↗

Reminder about uploaded revision

As the discussion period closes, we wanted to leave a brief final note. We have uploaded a revised PDF with all changes marked in blue, and our rebuttal above summarizes how it addresses your feedback.

In particular, we corrected Theorem 3, made the budget and cost accounting explicit, and clarified scale, oracle and novelty.

We would be grateful if you would take the revised version into account when finalizing your evaluation, and we hope you will consider raising your score. Thank you again for your time and for improving the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.