Operator-Valued Q-Learning: Compositional Bellman Recursions for Order-Sensitive Decision Making

In quantum pulse sequences, spacecraft attitude control and in-hand reorientation, actions act on a hidden physical state as non-commuting operators, so the outcome of a plan depends on the order of its actions. Standard value functions either observe this latent state and learn a nonlinear function of it, or record the action history, which in tabular form costs $K^T$ entries for $K$ actions and horizon $T$. We instead let the value function carry order: each $Q(s,a)$ is an $N\times N$ operator, and the Bellman backup applies the action's operator to the continuation value, $Q(s,a)=\text{Kd}a(R(s,a)+\gamma \mathbb{E}V(s^{\prime }))$. Unrolled along a plan, the recursion builds the ordered product of the actions' operators, and the value at any latent state is a linear trace readout. For policies that do not observe the latent, this representation is exact; the backup contracts for unitary and orthogonal dynamics, and an asynchronous operator temporal-difference (TD) scheme learns it with almost-sure convergence. Control needs a maximum over actions, which operators lack: by Kadison's anti-lattice theorem, self-adjoint operators generally have no least upper bound. An operator log-sum-exp instead gives one operator per state–action pair whose readout upper-bounds the optimal value at every latent state, reducing to soft $Q$-learning when actions commute. We also prove that scalar values on observations that ignore action order must confuse swapped histories, that exact history tables grow exponentially for free action groups but polynomially for nilpotent ones, and a Hankel-rank lower bound on the operator dimension. On random and $SO(3)$-pointing control tasks, operator $Q$-learning reaches a normalised score of $0.95$ within $100$ episodes and transfers without retraining to unseen initial states ($0.95$–$0.99$), where a recurrent baseline drops to $0.21$–$0.53$. A DQN that observes the latent eventually matches or exceeds it, consistent with the certificate's slack.