Do Less, Decide Better: Optimal Human Dispatching in AI-Assisted Decisions

AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments are often biased and incomplete. We study selective human oversight: given AI signals, which factors should be escalated to a costly human evaluator? We formulate this human–AI collaboration problem as a factor-level information-acquisition problem. Under squared loss, solving for optimal dispatching reduces to maximizing a contextual reward; under a linear model, this reward admits a closed-form decomposition into two interpretable terms: predictive relevance and residual uncertainty in the human evaluation after conditioning on the AI signal. We instantiate the framework on peer review, decomposing papers into ten aspects and evaluating 3,408 ICLR submissions across three LLMs and multiple regression heads. Outputs based on the optimal dispatching rules match full-human-review performance using only 2–3 queried aspects out of 10. The framework remains effective when the human signal comes from a single noisy reviewer and across review years, suggesting robustness to reviewer inconsistency and and temporal distribution shift. These results position principled dispatching as a practical foundation for scalable human–AI decision systems.