Universal Bellman Completeness Forces Lumpable State Partitions

Bellman completeness is a key assumption in fitted value methods and offline reinforcement learning because repeated Bellman updates must remain representable in a fixed value space. However, its universal form leaves the structure of a complete value space and a way to check it unclear. We study this converse for a finite Markov decision process (MDP) with at least two actions and a value space containing the constant functions. We prove that closure under every optimal Bellman update using rewards and continuation values from the space holds exactly when the value space consists of functions constant on the blocks of a unique state partition, and the transition kernel for every action is lumpable with respect to that partition. Independent action rewards expose closure under pointwise maxima and isolate each action's transition kernel. Together, these constructions yield an exact finite certificate based on adaptive Bellman membership queries. On a block-constant value space, they also give an exact relation between the closure defect and transition mismatch. Under separation or conditioning assumptions, this relation supports block recovery and yields value and policy loss bounds.