Cost-Aware Threshold Adaptive-K Learning to Defer under Budget Constraints

Learning to defer routes each input to a prediction source, trading accuracy against consultation cost. In practice several predictors and human experts are available, their costs differ widely, and existing multi-expert and top-$k$ methods regulate costs via a hyperparameter, yet they fail to guarantee strict adherence to budget constraints. We introduce CATABC, a budget-constrained adaptive top-k architecture that selects how many and which experts to consult per input, and weighs the consulted ones by masked attention over the image and expert features, under an average-cost constraint enforced by a dual price. We prove that the threshold selection optimally solves the Lagrangian of the average-budget proxy, that the price is the slope of the risk–budget frontier, and that the excess risk decomposes into pool, selection and fusion gaps. On CIFAR-10, CIFAR-100 and SVHN, CATABC stays within the prescribed budget and surpasses the baselines in most settings.