acceptodds
Under review as a conference paper at ICLR 2027

Operator-Valued Q-Learning: Compositional Bellman Recursions for Order-Sensitive Decision Making

Abstract

In quantum pulse sequences, spacecraft attitude control and in-hand reorientation, actions act on a hidden physical state as non-commuting operators, so the outcome of a plan depends on the order of its actions. Standard value functions either observe this latent state and learn a nonlinear function of it, or record the action history, which in tabular form costs entries for actions and horizon . We instead let the value function carry order: each is an operator, and the Bellman backup applies the action's operator to the continuation value, . Unrolled along a plan, the recursion builds the ordered product of the actions' operators, and the value at any latent state is a linear trace readout. For policies that do not observe the latent, this representation is exact; the backup contracts for unitary and orthogonal dynamics, and an asynchronous operator temporal-difference (TD) scheme learns it with almost-sure convergence. Control needs a maximum over actions, which operators lack: by Kadison's anti-lattice theorem, self-adjoint operators generally have no least upper bound. An operator log-sum-exp instead gives one operator per state–action pair whose readout upper-bounds the optimal value at every latent state, reducing to soft -learning when actions commute. We also prove that scalar values on observations that ignore action order must confuse swapped histories, that exact history tables grow exponentially for free action groups but polynomially for nilpotent ones, and a Hankel-rank lower bound on the operator dimension. On random and -pointing control tasks, operator -learning reaches a normalised score of within episodes and transfers without retraining to unseen initial states (–), where a recurrent baseline drops to –. A DQN that observes the latent eventually matches or exceeds it, consistent with the certificate's slack.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Q-learning, Non-Commutative Probability, Free Probability, Operator Algebras, Bellman Equation, Contraction Mapping, Order-Sensitive Markov decision processes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_muidOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses RL where actions act on a hidden state as non-commuting operators, so returns depend on action order. It represents each Q(s, a) as an N×N operator whose Bellman backup composes the actions’ operators along a plan, with the value at any latent state read out as a linear trace; for latent-blind policies this is exact, contractive under unitary/orthogonal dynamics, and learnable by operator TD with almost-sure convergence. Since self-adjoint operators lack a least upper bound, control uses an operator log-sum-exp that upper-bounds the optimal value and reduces to soft Q-learning when actions commute, alongside separation, complexity, and Hankel-rank results. On random and SO(3)-pointing tasks the method reaches 0.95 within 100 episodes and transfers to unseen initial states (0.95–0.99) where a recurrent baseline drops to 0.21–0.53, though a latent-observing DQN eventually matches it.

Critical strengths

The mathematics is believable and the writing is good. I would even say it is elegant.

The checks (E7, E9, E12, E17) are good practice: each theorem is exercised numerically.

The transfer result is convincing.

Critical weaknesses

W1 Theorem 1 is a linearity statement: linear reward, known linear latent dynamics, latent-blind policy, hence a value linear in the latent with the “operator Q” as its coefficient. This is the same fact behind α-vectors, OOMs, and PSRs, written as a Bellman equation. The authors should state precisely what is new beyond the packaging.

W2 The comparison is uneven. OpQ-soft is given the true operators K_a; DQN sees only ρ_t; DRQN sees only the history. The transfer result in particular—DRQN trained on one ρ_0, tested on unseen latents—follows from that asymmetry. Give the baselines K_a as features, or compare against planning with the known model.

W3 The latent can be simulated and the problem is planning. The paper needs a concrete case where the latent is hidden, the dynamics are known, and learning beats simulation; estimating R(m, a) from scalar returns is mentioned but not developed.

W4 Theorems 4–6 follow from standard results and should be presented as consequences, not contributions.

Questions for authors

Q1. Can you point to a concrete problem where the operators are available but the initial latent is not, and cannot be recovered from a handful of observations?

Q2. Would allowing a small number of operators per state-action, narrow the gap?

3 comments
AuthorsOpenReview ↗

Response to Reviewer muid

Thank you for the overall positive assessment and for the questions that strengthened the paper.

W1 (what is new beyond the packaging). The exact representation is the α\alpha-vector/OOM/PSR linearity principle, and it is now Proposition 1 with that attribution. The new content is control in a value space with no lattice join:

  • a certificate valid for every policy where QMDP/FIB have no analogue (Theorem 2, Remark 2);

  • the characterisation of what fails there (monotonicity, non-expansiveness, a pβNp_\beta\sqrt N Lipschitz lower bound);

  • new in the revision: Corollary 2, Proposition 4, finite-horizon convergence of the sampled control update (Theorem 3), and E20’s multi-operator family.

W2 (uneven comparison). We agree, and E19 now runs it with matched knowledge. On DRQN: its ρ0\rho_0 input is constant in fixed-regime training, so its transfer score is not informative; we removed it from the abstract and say so in Sec. 6. Raw Ka\mathcal K_a is constant per action, so we implemented your suggestion as DQN trained on Ka\mathcal K_a-generated transitions, the informative form. With it, DQN reaches OpQ’s transfer on SO(3)SO(3) and exceeds it on C1 and C3 (held-out 0.99/0.98/0.840.99/0.98/0.84 vs. 0.95/0.99/0.680.95/0.99/0.68), but only when its synthetic latents leave the training support (0.88/0.96/0.590.88/0.96/0.59 otherwise). OpQ obtains transfer with no augmentation. We also compare with planning on a learned model, which is more sample-efficient at short horizons but whose decision cost grows as KHK^H.

W3/Q1 (a concrete hidden-latent problem). Quantum control with a calibrated gate set and an unknown input state.

  • Measurement is destructive and an episode has one copy, so latent-aware policies cannot be realised.

  • Recovering ρ0\rho_0 needs Ω(dr/ε2)/log⁡(d/rε)\Omega(dr/\varepsilon^2)/\log(d/r\varepsilon) copies even with optimal entangled measurements (Haah et al., 2017), with d=2nd=2^n for nn qubits.

  • Only latent-blind policies are available there, and Proposition 1 gives their value at every unknown input from one operator: the Bayes value under a prior is τ(ρˉ0Q)\tau(\bar\rho_0Q), and the worst case over pure inputs is λmin⁡(Q)\lambda_{\min}(Q).

New experiment E23 (Sec. 6, App. M) tests this end to end. ρ0\rho_0 is drawn from a prior with known mean and never observed; agents track the belief mean ρˉt\bar\rho_t. By linearity the best value of any policy of the action and mode history is computed exactly.

  • OpQ trained with the latent and deployed on ρˉt\bar\rho_t is unchanged (0.96/0.98/0.790.96/0.98/0.79), because plugging in the mean is exact for linear values; a DQN goes from 0.97/0.91/1.000.97/0.91/1.00 to 0.85/0.95/0.450.85/0.95/0.45, losing most on C3.

  • With the reward observable known, OpQ learns fastest on C2/C3 (0.97/0.530.97/0.53 at 100 episodes vs. 0.64/−0.060.64/{-0.06}) and decides in 0.010.01 ms vs. 940940 ms for blind exhaustive search. A belief-mean DQN ends higher on C3 (1.001.00 vs. 0.790.79), and learning RR from belief means slows OpQ; we report both.

The exactness follows from Proposition 1, so it holds by construction; the experiment shows that it costs OpQ nothing and fails for a nonlinear value function. E23 assumes known operators and a known prior mean, and its policies do not condition on past rewards: on C1/C2 a reward-based Bayesian filter could do better, bounded by the latent-aware oracle, while on C3 the reward arrives only at the end. App. M states all three. When ρ0\rho_0 is known, the problem is planning with Ka\mathcal K_a known and PP, RR learned. E19 now includes this comparison, and E20 shows that OpQ’s certificate improves depth-limited planners as a leaf value on two-qubit preparation (held-out fidelity 0.72→0.850.72\to0.85 at depth 3 and 0.87→0.920.87\to0.92 at depth 5, H=12H{=}12).

W4 (Theorems 4–6). Agreed. They are now Propositions 8–10, attributed to Carlyle–Paz/Fliess, Epstein and the BCH formula. The C$^*$-algebraic refinement of the operator dimension (Proposition 13) remains as the non-standard part.

Q2 (a few operators per state-action). Yes, and it gives a clean bridge to α\alpha-vector methods. With point-based backups (E20), M=32M{=}32 reaches 1.001.00 on C1 and M=8M{=}8 reaches 1.001.00 on two-qubit preparation from the training latent (0.950.95 held-out at M=64M{=}64). On SO(3)SO(3) pointing the single LSE operator (0.990.99) is better than point-based sets up to M=64M{=}64 (0.980.98), so where the slack is small one operator suffices. The MM-operator family is point-based value iteration written with operators, and App. J now says we do not claim it as new. What changes is the upper side: point-based solvers that bracket V∗V^*, such as HSVI, take their upper bound from the fast-informed bound, a lattice join; here Theorem 2 supplies it. App. J also explains when one operator suffices, how to choose MM with the lower bound LML_M, and the cost; guarantees cover the exact-model construction only.

Dear Reviewer muid,

Thank you again for your positive assessment and for your questions. As the discussion period is coming to a close, we wanted to check whether our response addresses them, and whether anything else would be helpful while there is still time.

In brief: the new experiment E23 tests the hidden-initial-state setting you asked about end to end, with its assumptions stated in App. M; E19 runs the baseline comparison with matched knowledge, including the DQN variant you suggested; and App. J positions the multi-operator family as point-based value iteration in operator form, with Theorem 2 supplying its upper bound.

We would be glad to answer any further questions.

Dear Reviewer muid,

We understand this is a busy period and are grateful for your time. As the discussion window closes soon, we would welcome any reaction to our response, even a brief one.

The changes most relevant to your review are E23, which tests the hidden-initial-state setting end to end, and the matched-knowledge comparison in E19, which includes the DQN variant you suggested. We would be happy to clarify either.

Reviewer_V65nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Operator-valued Q-learning (OpQ) for RL problems where the order of actions affects the outcome. The authors use matrix-valued Q-functions and introduce an operator Bellman recursion to represent action order. Experiments on random operator dynamics, SO(3) control, and quantum state preparation compare OpQ with scalar Q-learning, DQN, and recurrent baselines.

Critical strengths
  1. The paper introduces an operator-valued Bellman recursion with theoretical results on exact policy evaluation and convergence.
  2. The experiments cover several non-commutative control tasks and show improvements in sample efficiency and transfer to unseen initial states.
Critical weaknesses
  1. OpQ uses known action operators in its Bellman updates, while DQN and DRQN do not have access to the same information. The experiments therefore do not show how much of the improvement in sample efficiency and generalization comes from the proposed Q representation. A comparison with methods using the same action dynamics is needed.
  2. The operator log-sum-exp provides an upper bound on the optimal value, but this does not guarantee optimal action selection. In the two-qubit experiments, OpQ performs worse than DQN even with the exact model. The paper does not explain under what conditions this approximation can achieve good control performance.
Questions for authors
  1. How would OpQ compare with methods that use the same known action operators? How much of the reported improvement comes from the operator-valued Q representation?
  2. Under what conditions does the operator log-sum-exp provide accurate action selection? How does the gap between the upper bound and the true optimal value affect control performance?
5 comments
AuthorsOpenReview ↗

Response to Reviewer V65n

Thank you for the constructive questions; both led to new results.

W1/Q1 (same action dynamics; source of the improvement). E19 gives every learner and planner the same knowledge as OpQ: DQN on Ka\mathcal K_a-generated transitions with OpQ’s reward estimate, linear-in-ρ\rho Q-learning without Ka\mathcal K_a, and exhaustive and UCT search on a learned model. Three findings:

  1. Linearity alone does not explain OpQ’s early scores: Linear-Q scores 0.77/0.57/0.250.77/0.57/0.25 against 0.95/0.95/0.580.95/0.95/0.58 at 100 episodes.

  2. Much of the early improvement over model-free DQN is knowledge of Ka\mathcal K_a. DQN with Ka\mathcal K_a data reaches 0.93/0.82/0.220.93/0.82/0.22, and OpQ remains ahead on C2 and C3.

  3. OpQ’s zero-shot transfer is structural: its update never reads ρ\rho. A DQN matches it only with Ka\mathcal K_a-generated latents outside the training support (held-out 0.99/0.98/0.840.99/0.98/0.84; 0.88/0.96/0.590.88/0.96/0.59 on-support), against 0.95/0.99/0.680.95/0.99/0.68 for OpQ without augmentation.

By 3000 episodes DQN with Ka\mathcal K_a data matches or exceeds OpQ (0.99/0.99/0.860.99/0.99/0.86 vs. 0.95/0.99/0.680.95/0.99/0.68), so we claim early performance and decision cost, not asymptotic superiority. All tables and controls are in Appendix H.

W2/Q2 (when does the LSE certificate give good actions?). We agree that an upper bound alone does not guarantee greedy optimality; Table 6 isolates exactly this gap. We added four results.

  • The learned operators reach the certificate (new Theorem 3). For a finite horizon the sampled update converges almost surely to QβQ^\beta. E24 confirms it: error ≤0.01\le0.01, and greedy scores equal those of the exact QβQ^\beta (0.952/0.986/0.6800.952/0.986/0.680). The gap to the optimum is therefore the certificate’s own, which the next three results address.

  • A sufficient condition (new Proposition 4). If the slack τ(ρQβ)−q∗\tau(\rho Q^\beta)-q^{\ast} varies across actions by εt\varepsilon_t in expectation along the greedy policy, the loss is at most ∑tγtεt\sum_t\gamma^t\varepsilon_t. What governs control is the spread of the slack across actions, not its size. On SO(3)SO(3) pointing the certificate is 1.3×V∗1.3\times V^{\ast} but its mean spread is 2%2\% of V∗V^{\ast}, and greedy control is within 0.020.02 of optimal. On C1 and C3 the spread is 88–15%15\%. The bound holds on every instance (E20).

  • The aggregator is a choice (Corollary 2). Theorem 2 holds for any Löwner upper bound. The trace-minimal SDP bound gives a tighter certificate on C1 (1.88→1.67×V∗1.88\to1.67\times V^{\ast}) but not better control, as Proposition 4 predicts.

  • A family that reaches the optimum. For finite horizons the optimal value is a maximum of finitely many linear functions of ρ\rho, so keeping MM operators per state converges to it:

MM1 (LSE)8163264
C1, training latent0.860.980.971.001.00
C3, training latent0.791.001.001.001.00
C3, held-out latents0.680.820.880.920.95

Unlike point-based POMDP solvers such as HSVI, which take this upper bound from a lattice join, the bracket here rests on Theorem 2. Proposition 4 is sufficient, not necessary, and the spread needs the model, so we present it as an explanation rather than an a-priori test. For choosing MM, every stored operator is the value of a conditional plan, so the family gives a lower bound LM≤V∗L_M\le V^{\ast} that, with the certificate, brackets V∗V^{\ast} (App. J; at the training latent L64L_{64} is 4.154.15 vs. V∗=4.17V^{\ast}=4.17 on C1, and L8=V∗=1L_8=V^{\ast}=1 on C3). The price is MM times the memory and decision cost (0.80.8–1.91.9 ms at M=64M{=}64 vs. <0.11<0.11 ms). The single-operator limitation on two-qubit preparation is stated prominently.

Dear Reviewer V65n,

Thank you again for your constructive questions. As the discussion period is coming to a close, we wanted to check whether our response answers them, and whether any questions remain open while there is still time to reply.

In brief: E19 gives every baseline and planner the same action operators as OpQ; new Theorem 3 and experiment E24 show that the learned operators converge to the certificate, so the remaining gap to the optimum is the certificate's own; and Proposition 4 and E20 explain when the certificate's greedy policy is good, with a multi-operator family that closes the gap.

We would be glad to provide any further detail, and grateful if you would take the revision into account in your final assessment.

Dear Reviewer V65n,

We understand this is a busy period and are grateful for your time. As the discussion window closes soon, we would welcome any reaction to our response, even a brief one.

The two changes most relevant to your review are E19, which gives every baseline and planner the same knowledge of the operators as OpQ, and Theorem 3 with E24, which show that the learned operators converge to the certificate. We would be happy to clarify either.

Reviewer_WV5jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on problems where actions act on a hidden state as non-commuting operators, so the order of a plan matters. In this scenario, each Q(s,a)Q(s,a) is an N×NN\times N operator with backup Q(s,a)=Ka†(R+γEV)Q(s,a)=\mathcal{K}_a^\dagger(R+\gamma\mathbb{E}V), which unrolls into the ordered product of action operators and gives the value at any latent state through a linear trace readout. This is exact for latent-blind policy evaluation, and operator TD converges. an operator log-sum-exp upper-bounds every policy's value and reduces to soft Q-learning when actions commute. Experiments on three small control tasks show fast learning and zero-shot transfer to unseen initial latents.

Critical strengths

S1. The exact, latent-linear representation (thm1,2) is clean. It makes zero-shot transfer to unseen initial latents work since one operator per (s,a)(s,a) holds for every ρ\rho.

S2. The LSE certificate (thm3) holds for every policy without monotonicity requirements and collapses to soft Q-learning / a QMDP-type bound in the commuting case. all claims are numerically checked.

Critical weaknesses

W1. Vectorize ρ\rho and Thm. 1 is basically blind-policy α-vector evaluation, e.g., ρ↔b\rho\leftrightarrow b, Ka†↔Ta⊤\mathcal{K}_a^\dagger\leftrightarrow T_a^\top, τ(ρQ)↔⟨b,α⟩\tau(\rho Q)\leftrightarrow\langle b,\alpha\rangle. The related work sec says the difference is that α-vectors go through a stochastic belief update, but in the no-information-observation setting thm1 needs, the belief update is linear and deterministic, Ta⊤T_a^\top is the dual of a known non-commuting map, and you get ordered products too. So putting order in the codomain is already in α-vectors.

W2. The KTK^T history-table motivation argues against something POMDP rarely use. We track the belief when model is known and compress histories when model-free. OpQ also needs Ka\mathcal{K}_a known, and once Ka\mathcal{K}_a and ρ0\rho_0 are known ρt\rho_t is computable, so belief-based methods also avoid KTK^T, as the authors' own DQN-on-ρt\rho_t does. The case where history tables are the only option (unknown Ka\mathcal{K}_a) is exactly the one OpQ doesn't handle.

Questions for authors

Q1. In POMDP terms, order sensitivity just means the belief depends on the ordered history and can't be compressed into action counts. That's the normal situation whenever transition matrices don't commute, and even classical stochastic ones usually don't. What do the authors think is specific to the operator setting here?

Q2. In E18, OpQ gets the known Ka†\mathcal{K}_a^\dagger while DQN and DRQN are model-free, so the sample-efficiency and transfer gains may come from model knowledge. How can you isolate the gain of the representation from this prior model knowledge, or compare OpQ against a model-based baseline with access to the same Ka\mathcal{K}_a?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WV5j

Thank you for a review that engages directly with the mathematics.

W1 (α\alpha-vectors). We agree. With ρ\rho vectorised, the exact representation is blind-policy α\alpha-vector evaluation in an N2N^2-dimensional representation. We have removed the sentence contrasting it with a “stochastic belief update”, and the result is now Proposition 1 with that attribution (Hauskrecht, 2000; see also Even-Dar et al., 2005; Barry et al., 2014). This linearity is the paper’s foundation, not its contribution. The contribution is control when the value codomain is non-commutative (Q1).

Q1 (what is specific to the operator setting). Order-sensitivity is indeed generic: any POMDP with non-commuting TaT_a has it. What is not generic is the order structure of the value space.

  • Classical α\alpha-vectors live in the commutative space RS\mathbb R^{\mathcal S}, ordered by a simplicial cone. This is a lattice, so a coordinatewise maximum exists, and QMDP and FIB are exactly that join, whether or not the TaT_a commute.

  • Operator values live under the PSD cone, which is not a lattice (Yudin; Aliprantis & Tourky, 2007, Thm. 3.21) and is in fact an anti-lattice (Kadison).

  • A 2×22\times2 example (now in Remark 2). Let two action values be x1=diag(1,0)x_1=\mathrm{diag}(1,0) and x2=diag(0,1)x_2=\mathrm{diag}(0,1). Both II and YY, with diagonal entries 22 and off-diagonal entries 1.21.2, are upper bounds (Y−xiY-x_i has eigenvalues 0.2,2.80.2, 2.8). II is minimal, since an upper bound Z⪯IZ\preceq I has Z11=Z22=1Z_{11}=Z_{22}=1 and zero off-diagonal, yet I⪯̸YI\not\preceq Y (Y−IY-I has eigenvalue −0.2-0.2). So no least upper bound exists, and no max-backup over these two actions. As α\alpha-vectors, (1,0)(1,0) and (0,1)(0,1) have the join (1,1)(1,1); restricted to diagonal operators, II is that join, which is exactly why Proposition 3 recovers QMDP.

  • There is therefore no max-backup, and the natural smooth substitute is neither monotone (E12: 72/200072/2000 violations, persisting at perturbation scale 10−410^{-4}) nor non-expansive (E7: ratio 1.481.48 at β=10\beta{=}10). For a finite horizon this does not prevent the sampled update from converging to the certificate (new Theorem 3).

Theorem 2 still certifies an upper bound for every policy, latent-aware and history-dependent ones included, and reduces exactly to soft QMDP in the commutative case (Proposition 3). A second difference is that unitary dynamics never forget order: E10 shows a constant per-depth commutator term (0.2210.221 for so(3)\mathfrak{so}(3)), whereas scrambling stochastic maps forget geometrically. Both points are now Remark 2, which also states that the distinction concerns the ordered value space, not whether transition operators commute; general POMDPs are not the commuting case of our formulation.

W2 (history tables). We agree that the KTK^T framing overstated the alternative. The introduction now says that, when Ka\mathcal K_a and ρ0\rho_0 are known, the latent can be tracked exactly, as a belief is, so the problem reduces to the observed case. Exact tabular summaries are now discussed only where they belong (Proposition 9). Two clarifications:

  1. OpQ needs Ka\mathcal K_a but not PP, and its headline variant learns RR.

  2. On unknown operators we claim only representational capacity. E8/E13–E16 fit an operator model to exact sequence returns by supervised regression and reach test MSE ≤3×10−30\le3\times10^{-30} at the predicted dimension, versus 0.0030.003–0.090.09 for GRU/LSTM. Identifying operators from realistic samples, and coupling this with TD under guarantees, remain open.

Q2 (isolating model knowledge). E19 addresses this as far as an experiment can; see the table in the general response.

  • Without the operators, the same linear representation scores 0.77/0.57/0.250.77/0.57/0.25 at 100 episodes against OpQ’s 0.95/0.95/0.580.95/0.95/0.58, so linearity alone does not explain the gain; the arms also differ in synthetic data and gradient steps.

  • With the operators, DQN closes much of the early gap (0.93/0.82/0.220.93/0.82/0.22; 0.96/0.90/0.380.96/0.90/0.38 with four updates per step), and OpQ stays ahead early on C2 and C3; by 3000 episodes DQN with Ka\mathcal K_a data matches or exceeds it.

  • A planner with OpQ’s knowledge is more sample-efficient at short horizons, which we now state in the paper. Its cost per decision grows as KHK^{H}; OpQ’s does not.

Dear Reviewer WV5j,

Thank you again for your review. As the discussion period is coming to a close, we wanted to check whether our response addresses your concerns and to ask whether anything remains unclear, while there is still time to answer.

In brief: the exact representation is now credited as operator α-vector evaluation (Proposition 1); Remark 2 gives a 2×2 example of why operator values have no lattice join, which is what separates our setting from POMDPs; new Theorem 3 shows that the sampled update nevertheless converges to the certificate for a finite horizon; and E19 compares against baselines and planners that have the same knowledge of the operators.

We would be glad to clarify any point, and grateful if you would consider these changes in your final assessment.

AuthorsOpenReview ↗

Your concerns addressed: a quick look before the discussion closes?

Dear Reviewer WV5j,

We understand this is a busy period and are grateful for your time. As the discussion window closes soon, we would welcome any reaction to our response, even a brief one.

The two changes most relevant to your review are the 2×2 example in Remark 2, showing why operator values have no lattice join, and the new Theorem 3, which shows that the sampled update converges to the certificate for a finite horizon. We would be happy to clarify either.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

Theorem, proposition, section, and appendix numbers refer to the revised PDF, in which all changed text is blue.

In brief.

  1. What is new: certification and control when the value space has no lattice join. diag(1,0)\mathrm{diag}(1,0) and diag(0,1)\mathrm{diag}(0,1) have no least Löwner upper bound, so no max-backup exists (Remark 2); Theorem 2 still certifies every policy.

  2. What we concede: the representation is operator α\alpha-vector evaluation (now Proposition 1), and much of the sample efficiency comes from knowing the operators (E19).

  3. What survives: zero-shot transfer without augmentation, exact action on the belief mean when the latent is hidden (E23), tree-search scores on SO(3)SO(3) pointing at about 104×10^4\times lower decision cost, and a certificate valid for every policy, to which the sampled control update provably converges for a finite horizon (new Theorem 3).

We thank the reviewers for careful and convergent reviews. All three judged the theory sound (muid: “elegant”), valued the numerical checks, and found the zero-shot result convincing. The revision is substantial (three new results, six new experiments E19–E24, relabelled theorems), but every change answers a specific request, and each reply below says which. The revised PDF is uploaded.

Why this matters. Every max-backup, and every QMDP or fast-informed bound, needs a lattice. α\alpha-vectors always have one, however non-commuting the transitions; self-adjoint operators under the PSD cone do not (Yudin; Kadison), and quantum and attitude control live there. The paper gives RL machinery that works without a lattice: an exact compositional backup, convergent operator TD for policy evaluation, and a certificate valid for every policy, which reduces to soft QMDP when the lattice is restored.

E19: knowledge of the operators. OpQ uses the action maps Ka\mathcal K_a, not the kernel PP, and learns RR. E19 crosses {nonlinear, linear in ρ\rho} with {no Ka\mathcal K_a, known Ka\mathcal K_a} and adds planners that know exactly what OpQ knows. Random initial latents, 100 episodes (C1/C2/C3):

Agentknows Ka\mathcal K_ascore after 100 episodes
OpQ-softyes0.95/0.95/0.58\mathbf{0.95/0.95/0.58}
DQN + Ka\mathcal K_a data (4 updates/step)yes0.96/0.90/0.380.96/0.90/0.38
DQN + Ka\mathcal K_a datayes0.93/0.82/0.220.93/0.82/0.22
DQN on ρt\rho_tno0.89/0.73/0.040.89/0.73/0.04
Linear-in-ρ\rho Qno0.77/0.57/0.250.77/0.57/0.25
  • Linearity alone does not explain OpQ’s early scores (Linear-Q). The arms also differ in synthetic data and gradient steps, so this is not an isolated test of the representation.

  • OpQ stays ahead early on C2 and C3, but much of its early advantage is knowledge of Ka\mathcal K_a, and by 3000 episodes DQN with Ka\mathcal K_a data matches or exceeds it (0.99/0.99/0.860.99/0.99/0.86 vs. 0.95/0.99/0.680.95/0.99/0.68).

  • Zero-shot transfer: OpQ 0.95/0.99/0.680.95/0.99/0.68 without augmentation; DQN with Ka\mathcal K_a data 0.99/0.98/0.840.99/0.98/0.84, but only with synthetic latents off the training support (0.88/0.96/0.590.88/0.96/0.59 otherwise).

  • At short horizons a planner is more sample-efficient than every learner. On SO(3)SO(3) at H=8H=8–2020, OpQ matches 1000-simulation UCT (0.980.98–0.990.99 vs. 0.960.96–0.980.98) at about 104×10^4\times lower decision cost.

The paper now claims a compositional inductive bias with low decision cost, not superior asymptotic control.

E20: from one operator to the optimum. Any Löwner upper bound certifies (Corollary 2); the SDP bound is tighter (C1: 1.88→1.67×V∗1.88\to1.67\times V^*) but not better for control, as Proposition 4 explains. A few operators per state close the gap: C1 0.86→1.000.86\to1.00 at M=32M{=}32, C3 0.79→1.000.79\to1.00 at M=8M{=}8 (0.950.95 held-out at M=64M{=}64).

Control convergence (new Theorem 3, E24). For a finite horizon, the sampled OpQ-soft update converges almost surely to the certificate QβQ^\beta, by backward induction on tt; this needs no monotonicity or non-expansiveness. In E24 the learned operators come within 0.010.01 of QβQ^\beta and their greedy scores equal the exact certificate’s on C1–C3. The limit is QβQ^\beta, not Q∗Q^*, and the infinite horizon remains open.

Scope and robustness (App. J–N). A table separates what is proved. With miscalibrated gates (E21), OpQ on SO(3)SO(3) is as robust as planning, but on C1/C3 less so (C3: 0.79→0.490.79\to0.49 at ε=0.03\varepsilon=0.03), a stated limitation. E23 tests a hidden initial latent end to end: OpQ trained with the latent acts exactly on the belief mean (0.96/0.98/0.790.96/0.98/0.79), while a DQN goes from 0.97/0.91/1.000.97/0.91/1.00 to 0.85/0.95/0.450.85/0.95/0.45.

AuthorsOpenReview ↗

Summary of revisions (blue in the PDF)

  1. Positioning. New Remark 2, with a 2×22\times2 example of two operators with no least upper bound, and rewritten introduction and related work: the lattice/anti-lattice distinction, the α\alpha-vector attribution, latent tracking instead of history tables as the relevant alternative, and new references (Aliprantis & Tourky; Barry et al.; Even-Dar et al.; Kocsis & Szepesvári; Pineau et al.; Smith & Simmons; Sutton; Khaneja et al.). App. J states that the MM-operator family is point-based value iteration in operator form and that the certificate supplies its upper bound.

  2. Results relabelled. The exact representation is Proposition 1; the former Theorems 4–6 are Propositions 8–10.

  3. New theory. Corollary 2 (any Löwner upper bound certifies), Proposition 4 (slack spread governs greedy control), and Theorem 3 (the sampled OpQ-soft update converges almost surely to the certificate for a finite horizon), with proofs in Appendix B.

  4. New experiments. E19 (Sec. 6 and Appendix H): the 2×22\times2 study, all controls, transfer, planners, horizons up to 2020, and cost. E20 (Appendix I): aggregators, MM operators, slack spread, leaf value. E21–E24 (App. K–N): miscalibrated gates, sensitivity to β\beta, a hidden initial latent, and convergence of the sampled update.

  5. Corrections to E18. The history-table failure in the random regime is attributed to its key omitting ρ0\rho_0, not to Proposition 8. The DRQN transfer figure is removed from the abstract. The learned-RR score on C3 is noted to be within one standard deviation of the exact-model ceiling. The protocol now states what each agent knows.

  6. Scope and limits. New App. J (what is proved and under which conditions; one versus several operators; choosing MM; cost) and E21 (App. K: miscalibrated gates; MM-operator lower bound and cost). The abstract, introduction and conclusion state what converges (policy evaluation; for a finite horizon, the sampled control update, to the certificate rather than Q∗Q^*) and that a DQN with the operators can match OpQ. App. M states that E23’s plug-in exactness holds by construction, that E23 assumes known operators and prior mean, and that its policies do not condition on past rewards. Sec. 5 and App. J state that memory and cost grow exponentially in the number of qubits (N=2nN=2^n; experiments use N≤4N\le4). Clarified: positivity of the LSE argument, that non-monotonicity affects the backup but not the scalar readout, and that E8/E13–E16 are supervised; GRAPE added to related work; Figure 1 labels updated.

  7. Space. The finite-instance checks (E7–E17) and the representation results are summarised in the main text and given in full in Appendix G; the main text is within 10 pages.

  8. Code. The anonymized repository now contains the code and raw results of E19–E24 (run_e19.py, run_rebuttal.sh); Section 6 links it again, App. O gives run times, and the Table 4 caption explains that experiment IDs are fixed labels shared with the code, hence not consecutive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.