- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses RL where actions act on a hidden state as non-commuting operators, so returns depend on action order. It represents each Q(s, a) as an N×N operator whose Bellman backup composes the actions’ operators along a plan, with the value at any latent state read out as a linear trace; for latent-blind policies this is exact, contractive under unitary/orthogonal dynamics, and learnable by operator TD with almost-sure convergence. Since self-adjoint operators lack a least upper bound, control uses an operator log-sum-exp that upper-bounds the optimal value and reduces to soft Q-learning when actions commute, alongside separation, complexity, and Hankel-rank results. On random and SO(3)-pointing tasks the method reaches 0.95 within 100 episodes and transfers to unseen initial states (0.95–0.99) where a recurrent baseline drops to 0.21–0.53, though a latent-observing DQN eventually matches it.
Critical strengths
The mathematics is believable and the writing is good. I would even say it is elegant.
The checks (E7, E9, E12, E17) are good practice: each theorem is exercised numerically.
The transfer result is convincing.
Critical weaknesses
W1 Theorem 1 is a linearity statement: linear reward, known linear latent dynamics, latent-blind policy, hence a value linear in the latent with the “operator Q” as its coefficient. This is the same fact behind α-vectors, OOMs, and PSRs, written as a Bellman equation. The authors should state precisely what is new beyond the packaging.
W2 The comparison is uneven. OpQ-soft is given the true operators K_a; DQN sees only ρ_t; DRQN sees only the history. The transfer result in particular—DRQN trained on one ρ_0, tested on unseen latents—follows from that asymmetry. Give the baselines K_a as features, or compare against planning with the known model.
W3 The latent can be simulated and the problem is planning. The paper needs a concrete case where the latent is hidden, the dynamics are known, and learning beats simulation; estimating R(m, a) from scalar returns is mentioned but not developed.
W4 Theorems 4–6 follow from standard results and should be presented as consequences, not contributions.
Questions for authors
Q1. Can you point to a concrete problem where the operators are available but the initial latent is not, and cannot be recovered from a handful of observations?
Q2. Would allowing a small number of operators per state-action, narrow the gap?
3 comments
Dear Reviewer muid,
Thank you again for your positive assessment and for your questions. As the discussion period is coming to a close, we wanted to check whether our response addresses them, and whether anything else would be helpful while there is still time.
In brief: the new experiment E23 tests the hidden-initial-state setting you asked about end to end, with its assumptions stated in App. M; E19 runs the baseline comparison with matched knowledge, including the DQN variant you suggested; and App. J positions the multi-operator family as point-based value iteration in operator form, with Theorem 2 supplying its upper bound.
We would be glad to answer any further questions.
Dear Reviewer muid,
We understand this is a busy period and are grateful for your time. As the discussion window closes soon, we would welcome any reaction to our response, even a brief one.
The changes most relevant to your review are E23, which tests the hidden-initial-state setting end to end, and the matched-knowledge comparison in E19, which includes the DQN variant you suggested. We would be happy to clarify either.
Response to Reviewer muid
Thank you for the overall positive assessment and for the questions that strengthened the paper.
W1 (what is new beyond the packaging). The exact representation is the α-vector/OOM/PSR linearity principle, and it is now Proposition 1 with that attribution. The new content is control in a value space with no lattice join:
a certificate valid for every policy where QMDP/FIB have no analogue (Theorem 2, Remark 2);
the characterisation of what fails there (monotonicity, non-expansiveness, a pβN Lipschitz lower bound);
new in the revision: Corollary 2, Proposition 4, finite-horizon convergence of the sampled control update (Theorem 3), and E20’s multi-operator family.
W2 (uneven comparison). We agree, and E19 now runs it with matched knowledge. On DRQN: its ρ0 input is constant in fixed-regime training, so its transfer score is not informative; we removed it from the abstract and say so in Sec. 6. Raw Ka is constant per action, so we implemented your suggestion as DQN trained on Ka-generated transitions, the informative form. With it, DQN reaches OpQ’s transfer on SO(3) and exceeds it on C1 and C3 (held-out 0.99/0.98/0.84 vs. 0.95/0.99/0.68), but only when its synthetic latents leave the training support (0.88/0.96/0.59 otherwise). OpQ obtains transfer with no augmentation. We also compare with planning on a learned model, which is more sample-efficient at short horizons but whose decision cost grows as KH.
W3/Q1 (a concrete hidden-latent problem). Quantum control with a calibrated gate set and an unknown input state.
Measurement is destructive and an episode has one copy, so latent-aware policies cannot be realised.
Recovering ρ0 needs Ω(dr/ε2)/log(d/rε) copies even with optimal entangled measurements (Haah et al., 2017), with d=2n for n qubits.
Only latent-blind policies are available there, and Proposition 1 gives their value at every unknown input from one operator: the Bayes value under a prior is τ(ρˉ0Q), and the worst case over pure inputs is λmin(Q).
New experiment E23 (Sec. 6, App. M) tests this end to end. ρ0 is drawn from a prior with known mean and never observed; agents track the belief mean ρˉt. By linearity the best value of any policy of the action and mode history is computed exactly.
OpQ trained with the latent and deployed on ρˉt is unchanged (0.96/0.98/0.79), because plugging in the mean is exact for linear values; a DQN goes from 0.97/0.91/1.00 to 0.85/0.95/0.45, losing most on C3.
With the reward observable known, OpQ learns fastest on C2/C3 (0.97/0.53 at 100 episodes vs. 0.64/−0.06) and decides in 0.01 ms vs. 940 ms for blind exhaustive search. A belief-mean DQN ends higher on C3 (1.00 vs. 0.79), and learning R from belief means slows OpQ; we report both.
The exactness follows from Proposition 1, so it holds by construction; the experiment shows that it costs OpQ nothing and fails for a nonlinear value function. E23 assumes known operators and a known prior mean, and its policies do not condition on past rewards: on C1/C2 a reward-based Bayesian filter could do better, bounded by the latent-aware oracle, while on C3 the reward arrives only at the end. App. M states all three. When ρ0 is known, the problem is planning with Ka known and P, R learned. E19 now includes this comparison, and E20 shows that OpQ’s certificate improves depth-limited planners as a leaf value on two-qubit preparation (held-out fidelity 0.72→0.85 at depth 3 and 0.87→0.92 at depth 5, H=12).
W4 (Theorems 4–6). Agreed. They are now Propositions 8–10, attributed to Carlyle–Paz/Fliess, Epstein and the BCH formula. The C$^*$-algebraic refinement of the operator dimension (Proposition 13) remains as the non-standard part.
Q2 (a few operators per state-action). Yes, and it gives a clean bridge to α-vector methods. With point-based backups (E20), M=32 reaches 1.00 on C1 and M=8 reaches 1.00 on two-qubit preparation from the training latent (0.95 held-out at M=64). On SO(3) pointing the single LSE operator (0.99) is better than point-based sets up to M=64 (0.98), so where the slack is small one operator suffices. The M-operator family is point-based value iteration written with operators, and App. J now says we do not claim it as new. What changes is the upper side: point-based solvers that bracket V∗, such as HSVI, take their upper bound from the fast-informed bound, a lattice join; here Theorem 2 supplies it. App. J also explains when one operator suffices, how to choose M with the lower bound LM, and the cost; guarantees cover the exact-model construction only.