Opportunity, Recoverability, and Hardware Realization in Conditional Neural Computation

Conditional computation assigns different amounts of computation to different inputs. Poor performance can have several causes. The candidate models may not make complementary errors, the router may choose poorly, or dynamic execution may erase the expected computational savings. We study these effects under a common expected-compute budget. Our framework measures oracle opportunity relative to a matched static model and compares it with an input-independent mixture control. It also measures how much of that opportunity a learned router recovers and provides an exact finite-sample accuracy oracle. We test the framework in three settings: a nested CNN on CIFAR-10, independently trained ResNet-18 candidates on CIFAR-100, and an MLP bank on CoverType. Candidate and router training use disjoint data splits. All three candidate banks show substantial oracle opportunity. Recovering it is harder. The prespecified marginal-utility router falls below the static reference on both image tasks and gives only a small gain on CoverType. In a secondary exploratory analysis, a ResNet margin policy frozen on validation data recovers about \(10%\) of the available opportunity. The hardware results reveal a separate limitation. On an RTX 3090, matched MACs do not imply matched latency or energy. The dynamic policy is slower and more energy-consuming than the static medium model, while its comparison with the static large model changes with batch size. Our results prove that we should therefore report candidate opportunity, routing recovery, and hardware realization as separate quantities.