Procedural Adversarial Training

For many computer vision tasks, neural networks can be trained to produce a low average error, yet they still fail on seemingly simple scenes. Reducing these failures requires first finding them, but they may not be in a fixed dataset. We propose procedural adversarial training (PAT), a minimax framework in which a learner trains a model while an adversary searches the input space of a procedural generator. The procedural generator is an interpretable program mapping a low-dimensional vector of control parameters (e.g. layout, lighting, pose) to a rendered image and its label. We introduce the Simple-PAT algorithm, which uses zeroth-order optimization in the procedural generator's parameter space to propose adversarial training samples that are generated on demand. We also create PAT-Dojo, a collection of efficient procedural generators for surface normal estimation, chair segmentation, and human pose estimation. Against a cross-model black-box adversary with 50M queries, Simple-PAT reduces worst case error by a geometric mean of $2.09\times$ compared to standard training techniques.