PACE: Cost-Efficient LLM Judging with Human-Agreement Control via Pre-Routing

LLM-as-a-Judge offers a scalable alternative to human evaluation, but disagreement with human preferences remains a central reliability challenge, and improving agreement through stronger Judges or repeated evaluations can be costly. We introduce PACE (Pre-routing with Agreement-Controlled Evaluation), a selective-evaluation framework that routes each instance to a single Judge tier before evaluation. Using labeled calibration data, PACE estimates each verdict's disagreement risk within its routing group, then pools the verdicts and reports the largest subset with controlled overall disagreement risk. Routing groups may be learned from the unlabeled test batch, introducing dependencies that standard fixed-group analyses do not address. Under leave-one-out routing stability and regularity conditions, we quantify the effect of routing instability on risk estimation and establish asymptotic mFDR control at level $\alpha$ for the reported verdicts. In comparisons with cascades and other baselines, PACE yields competitive agreement–coverage trade-offs with one evaluation record per instance while substantially reducing cost.