In-Context Cascade: Few-Shot Adaptation for Budget-Constrained Labeling at Scale

Labeling large pools of unlabeled text items with respect to a natural language predicate is a common bottleneck in data annotation, content moderation, and enterprise AI pipelines. While large language models can automate this task, exhaustive labeling by a costly oracle such as a frontier model or a human annotator is infeasible for pools of thousands to millions of items. We propose the In-Context Cascade (ICC), a budget-constrained pool labeling framework that combines in-context learning (ICL) adaptation of a cheap model with targeted oracle escalation. ICC allocates a small harvest budget $B_h$ from total budget $B$ to oracle-label items near the decision boundary, uses these as ICL demonstrations to adapt the cheap model across the remaining pool, and reserves an escalation budget $B_e=B-B_h$ to resolve residual uncertainty. We provide theoretical guarantees on coverage bounds, harvest strategy comparison, optimal budget allocation, and escalation thresholds, leveraging these for the algorithm. Experiments on four datasets with oracle budget $B$ from $1\%$ to $20\%$ of pool size $N$ show that ICC consistently outperforms uncertainty-based cascade baselines, with the largest gains often at the tightest budgets where ICL adaptation is most valuable.