Changing the Oracle While Holding Models Fixed Reveals Task-Specific Signal in Simulation-Assisted Ranking

Selecting a model with scarce task-native labels requires evidence that follows the target ranking, including candidate reversals that contradict familiar leaderboard orderings. We introduce fixed-candidate oracle intervention, a controlled validation framework that freezes the model pool and changes the public task-native scoring oracle. The framework measures oracle-shift recovery (OSR), agreement on candidate pairs whose task order reverses the reference ordering, making task specificity a direct empirical target. We instantiate the framework with ProbeAlign, which learns task geometry from probe responses on a real fit split, admits and weights generated items in that behavior space, tunes on a disjoint calibration split, and reads the holdout once. At $n=200$, ProbeAlign obtains regret $0.108/0.094/0.112/0.112$ and Kendall $\tau =0.71/0.69/0.68/0.68$ on LiveCodeBench-Lite, MMLU-Pro-Core, IFEval-Strict, and HELM-Lite, improving on bootstrap ($0.172/0.158/0.178/0.169$), Active Labeling (+$50$) ($0.138/0.127/0.144/0.133$), and IRT-on-real-only ($0.123/0.139/0.128/0.128$). Across $500$ paired trials against the strongest primary comparator, every regret interval excludes zero, OSR rises on all four interventions, and the selected winner improves by $51$–$74$ trials. Broader checks spanning two-oracle mechanism isolation, three simulator families, a complete second-pool replication, a $12$-model six-family pool, and four-task active-label frontiers retain positive aggregate improvements with task-dependent paired uncertainty. Comparable active rankings require 100–200 additional task-native labels. Oracle intervention therefore turns task specificity into an inspectable validation target, while probe-behavior calibration supplies decision-relevant ranking signal in the measured low-resource regime.