Minimal Complete Specification Languages via Specification Equivalence Backups

As AI systems become increasingly able to solve any task we can specify, the bottleneck becomes being able to more efficiently and robustly specify our tasks. In this work, we move towards the theoretical limit of optimal specification: finding the minimum specification language which allows for precise specification of any task in a domain. We formalize the Specification Language Design Problem, and define the minimum amount of information necessary to determine an optimal policy for the specification, which we call the Specification Equivalence Set. This set has the recursive substructure necessary for a Bellman backup to compute it exactly, allowing us to find optimal specification language efficiently and exactly for a set of specification problems. We then extend these insights to a deep RL algorithm, using a VQVAE-inspired method to train an agent that is controllable via a latent specification language. We find that this deep RL algorithm matches our exact algorithm at small scales, and allows the system to scale to larger domains. In introducing this problem and our methods of addressing it, we aim to provide a primary look into the benefits of more precise specifications for AI systems.