KOLO: Executable Challenge Bases for Neural Training

Standard diagnostics for monitoring neural-network training, such as loss curves, validation accuracy, and gradient norms, are descriptive rather than constructive. They fail to answer whether a superior model remains accessible within the same architecture and objective. We address this by studying executable challenges, which preserve selected intermediate representations of a trained network and reconstruct or re-optimize the remaining computation to evaluate the resulting complete model under the original objective. While monitoring $P$ representations induces an exponential lattice of $2^P$ possible retention patterns, we introduce KOLO (Keep One, Leave One): a linear-size family of just $2P+1$ challenges consisting of route zero, every single retained representation, and every single released representation. We show that, without structural assumptions, superior models can remain hidden in untested routes, but KOLO becomes a valid measurement basis when governed by a coherent complete-model compiler and bounded higher-order interactions. Theoretically, we characterize when pairwise interactions are exactly identified by KOLO—proving exact identification on trees and odd-unicyclic graphs—and geometrically distinguish route reconstruction (requiring linear-span coverage) from direct threshold certification (requiring convex-hull coverage). Operationally, Keep-One separates representation deficit from conditional optimization slack, while Leave-One can reveal more accessible finite-budget optimization paths. Finally, reconstructed route evidence transfers to a global suboptimality bound through explicit solver, comparison-class, and coverage defects. Empirically, on a trained CIFAR-10 network with $P=8$, KOLO evaluates only $17$ of $256$ routes yet captures an average of $95.6$ $\%$ of the exhaustive optimization headroom in the task-oriented challenge regime, retrieving the exact best route in $7$ of $9$ checkpoint–seed cases. These results establish executable challenge bases as a principled way to monitor not only what training achieved, but what better solutions remain constructively accessible.