acceptodds
Under review as a conference paper at ICLR 2027

GRPO-QPS: Target-Preserving Reinforcement Learning for Quantum Posterior Sampling

Abstract

Bayesian quantum tomography requires efficient inference while preserving a posterior fixed by the prior and Born likelihood. Learned transport provides fast amortized samples, but reward tuning can reshape the generated distribution rather than improve exploration of this fixed target. We introduce GRPO-QPS, a target-preserving framework in which GRPO learns proposal behavior and an exact Metropolis correction preserves the posterior after training. Across the evaluated reconstruction benchmarks, GRPO-QPS improves over BuresTomFlow and Flow-GRPO on thermal, cat, Dicke, and cluster families, and it closely matches an exact two-qubit reference posterior. Tuned conventional MCMC is slightly stronger on several original continuous benchmarks where the available fixed proposals already match the posterior geometry well. To test whether this reflects a fundamental limitation of learned exploration, we evaluate a more challenging multimodal thermal posterior. At six qubits and 800 shots, the learned proposal achieves a minimum effective sample size of 102 per likelihood calls, compared with 28 for prior independence, 27 for a tuned fixed mixture, and 20 for Haario adaptive Metropolis. A record-conditioned policy also transfers to unseen 3,000-shot records, matching or exceeding the strongest conventional baseline in all nine held-out seed-record comparisons. These results show that GRPO-QPS combines target-preserving Bayesian inference with broad gains over learned transport baselines and a sampling advantage when efficient exploration requires proposal geometry beyond the evaluated conventional kernels.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Quantum Tomography, GRPO, Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r1p3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a framework, GRPO-QPS, for quantum tomography, i.e., to reconstruct the density matrix (or its distribution) from a limited number of observations. It uses a learning-based sampler, along with Markov Chain Monte Carlo (MCMC), for faithful inference. The Metropolis correction is proposed in the paper. Experiments compare the proposed approach with conventional baselines Flow-GRPO (a general flow-based sampler) and BuresTomFlow.

Critical strengths

While other papers propose independent sampling methods, this paper proposes a Markov chain proposal model. I think being able to apply the Metropolis algorithm to flow-based models is a difficult problem because flow models do not give exact likelihoods. The paper claims to have solved this problem.

Critical weaknesses

The paper is very difficult to comprehend, partly because much knowledge of the application field (quantum computers) is assumed. Even otherwise, terms are used without being defined. This applies to all sections, including the method and experiments. Even after going through related papers, I found it difficult to understand the sentences in this paper.

The claims of the paper need to be supported by comparisons with relevant baselines, especially other established methods for Quantum Tomography (such as GQNQ [Zhu et al., 2022]). The paper uses BuresTomFlow as a baseline, but that itself is not available.

Questions for authors

Sec 3.1:

  • Is the target distribution known? Eq. (5) also uses πX\pi_X; what is it?
  • Line 127: What is the simulated truth? I assume the posterior is a distribution over ρ\rho. Do you mean the true ρ\rho is the truth?

Sec 3.2:

  • Line 157: what is purification factor?
  • Line 161: What is the declared posterior? Is it known? If not, how is it obtained?

Sec 3.3:

  • How is the end point ρg\rho_g obtained?
  • What is finite posterior comparator? Is it a set of samples ρ\rho from the learned-model?
  • Is the energy score, maximised instead of minimised? Then only can the said objectives in line 169 be achieved.
  • How are the trajectories of flow-model perturbed? What is group advantage, and what is the penalty mentioned in line 172? I can see these terms are borrowed from the Flow-GRPO paper, but explaining them here is necessary in the context.

Sec 3.4:

  • What is QϕQ_\phi? How is it implemented and trained? I assume xx is ρ\rho here, right? What is yy?
  • Many terms are used without context and are difficult to follow. E.g., enumerated-diagnostics (line 191), multimodal study (line 200).
  • How are the Gaussian mixture parameters of Eq. (7) learned? Do we have data of true trajectories?

Sec 4:

  • Please explain what Dicke, cluster, cat, random mixed-state problems and transverse-field Ising model are. Is the target distribution available for these? If yes, what is it?
2 comments

We thank the reviewer for a careful reading of a dense paper. We agree with the main criticism: the paper assumed familiarity with quantum tomography and used several terms before defining them. The revised version opens Section 3 with a notation table that defines these terms and gives the formula of every state family. We answer the questions below in the order in which they were asked. Section, table and figure numbers refer to the submitted version; the revision keeps them.

Sec. 3.1: Is the target distribution known, and what is ρ? ρ is a density matrix, the standard description of a quantum state: for n qubits, a positive semidefinite 2ⁿ × 2ⁿ matrix with unit trace. A measurement record y lists, for each measured copy of the state, the measurement setting and the observed outcome. By the Born rule, outcome m occurs with probability tr(E_m ρ). The target is the Bayesian posterior of Eq. (1): the prior multiplied by the probability of the observed record, then normalized. It is known up to its normalizing constant, so it can be evaluated at any state but not sampled directly, which is exactly the setting of Metropolis–Hastings. For the smallest problems (the two-qubit study of Table 3, the 48-state example and the multimodal study) we also compute the normalized posterior exactly, by numerical integration or enumeration, and use it as a reference.

Line 127: What is the simulated truth? In each experiment we draw a state ρ⋆ from the prior and simulate a measurement record from it. The methods never see ρ⋆; it is used only to score an estimate by its Bures distance to ρ⋆. The "truth" is therefore indeed the true ρ.

Sec. 3.2: What is a purification factor, and what is the declared posterior? Any density matrix of rank at most r can be written as ρ = FF†/tr(FF†), where F is a complex 2ⁿ × r matrix that we call the factor. Working with F guarantees that every state is positive semidefinite with unit trace. By the "declared posterior" we mean the posterior of Eq. (1) for the prior stated in Appendix C (Table 7); nothing in it is estimated from data.

Sec. 3.3: Flow-GRPO. BuresTomFlow defines an ordinary differential equation (Eq. 3) that carries a random initial state to a state consistent with the record. Flow-GRPO replaces it with a stochastic version that has the same marginal distributions: Gaussian noise is added at each integration step, so every step has a known Gaussian transition density. ρ_g is the state at t = 1 of the g-th such trajectory.

The finite posterior comparator {ρ̃_j} consists of M states obtained by importance resampling: candidates are drawn from the prior, weighted by their likelihood and resampled. They are not samples from the learned model. The energy score R_g is maximized. Its first term rewards closeness to the comparator, and its second term rewards spread among the G trajectories that share a record. The advantage of a trajectory is its reward minus the group mean, divided by the group standard deviation. The penalty is the Kullback–Leibler divergence between the updated and the original per-step transition densities, and it keeps the fine-tuned model close to BuresTomFlow. The revision states all of this in Section 3.3.

Sec. 3.4: The proposal Q_φ. Q_φ(x′ | x, y) is the proposal distribution of the Metropolis–Hastings sampler: given the current parameter x and the record y, it proposes a new parameter x′. The parameter determines the state, ρ = ρ(x). It consists of a few physical coordinates for the structured families and of the factor F for random mixed states.

We use two forms of Q_φ. The first chooses among four fixed moves (three step sizes and a fresh draw from the prior), with probabilities that depend on the state through a small linear layer. The second is a neural network that outputs the weights, shifts and covariances of a Gaussian mixture (Eq. 7). Training needs neither true states nor true trajectories. The sampler is simulated with the current proposal, and each trajectory is scored by how closely its averages match reference moments estimated from separate pilot chains. GRPO then raises the probability of the proposals made in trajectories that scored above their group's average.

The "enumerated diagnostics" are a study of a 48-state posterior whose transition matrix can be written down exactly (Appendix K). The "multimodal study" is the thermal experiment of Section 4.4, whose posterior has two mirror-image modes.

Sec. 4: The state families. All are standard test families in quantum information, and each structured family has a small noise parameter with a uniform prior. The revision gives each formula in Section 3:

FamilyDefinition
Cat (GHZ)(1 − γ)|C_φ⟩⟨C_φ| + (γ/2)(|0…0⟩⟨0…0| + |1…1⟩⟨1…1|), with |C_φ⟩ = (|0…0⟩ + e^{iφ}|1…1⟩)/√2
Dicke(1 − p)|D⟩⟨D| + pσ, where |D⟩ is the equal superposition of all basis states with two excitations and σ is a uniform mixture of such basis states
Cluster(1 − p)|C⟩⟨C| + (p/n) Σ_i Z_i|C⟩⟨C|Z_i, where |C⟩ is obtained by applying controlled-Z gates between neighbouring qubits of |+⟩^{⊗n}
Random mixedρ(F) for a complex Gaussian factor F of rank 16
Thermal Isingρ ∝ exp(−β(J Σ Z_iZ_{i+1} + h Σ X_i)), with coupling J, field h and inverse temperature β

As explained above, each posterior is available up to its normalizing constant, and exact references exist for the two-qubit study, the 48-state example and the multimodal study.

Comparison with established tomography methods, including GQNQ. Table 25 already compares the following methods on the same records:

  • maximum likelihood;
  • classical shadows;
  • a restricted Boltzmann machine density operator;
  • an amortized neural operator;
  • the prior mean.

Section 4.3 and Appendix I also include sequential Monte Carlo, the standard sampler for Bayesian tomography (Granade et al., 2016).

No public implementation of GQNQ is available, so we reimplemented its design. A representation network averages encodings of the measured settings and outcomes, and a generation network predicts the outcome distribution of any queried measurement. We trained it with five seeds on simulated records from the same families. We then recovered a state by linear inversion of its predicted distributions over all Pauli settings, followed by projection onto density matrices. Given exact probabilities, this procedure recovers the true state to a Bures distance of about 10⁻⁷. On the settings of Table 25, GQNQ obtains the following Bures distances to the true state (lower is better):

Family4 qubits6 qubits
Dicke0.4350.712
Cluster0.6020.723
Cat0.6430.591
Random mixed0.7681.073

GQNQ improves on the classical shadow and the RBM density operator in 7 of the 8 settings. It is less accurate than the amortized neural operator in all six structured settings, and less accurate than maximum likelihood in five of them. It also remains far from the prior mean and the Bayesian pipeline: on four-qubit Dicke states, for example, it reaches 0.435 against 0.041 for GRPO-QPS. With only eight measurement outcomes, a generic amortized model trained on the same families does not replace the physical prior. The revision adds GQNQ as a column of Table 25 and describes the implementation in Appendix O.

Availability of BuresTomFlow. BuresTomFlow is available at https://www.preprints.org/manuscript/202611.1144, for your reference

Reviewer_ysh5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at Bayesian quantum state tomography (QST), where the target is the posterior ΠX(dx∣y)\Pi_X(dx \mid y) and where reward-tuned generators (such as Flow-GRPO) can change the distribution they sample which is obviously an issue if one needs understand the system and proceed to Quantum Process Tomography or Quantum System Identification. So, to cure this issue the authors use GRPO only to learn a proposal policy Qϕ(x′∣x,y)Q_\phi(x' \mid x, y), freeze it, and put it inside a Metropolis-Hastings kernel with exact forward and reverse densities. Then the authores proceed to study two proposal classes : The first is a restricted scheduler that picks pCN or local step scales and it is used for the reconstruction and enumerated studies. The second is a state-dependent Gaussian mixture model with learned weights, drift and Cholesky factors used on a thermal TFIM posterior with two mirror modes in. Prop. 1 gives stationarity for a frozen, positive policy, and Prop. 2 gives convergence on a 48 state graph. As far as the empirics the paper reports large Bures error reductions over BuresTomFlow and Flow-GRPO (as seen in Fig. 2), near-exact agreement with a two-qubit reference posterior, and parity or losses against tuned MCMC on the original benchmarks reporting overall as main positive result a 102 vs 28 minimum ESS per 1,000 likelihood calls on a 6 qubit multimodal posterior.

Critical strengths

The main design choice is correct and clearly argued although I do think that other schemes, e.g. QST using polynomial optimization are way more trustworthy. In this context though, learning only the proposal and keeping the full MH ratio, with both Qϕ(x′∣x,y)Q_\phi(x' \mid x, y) and Qϕ(x∣x′,y)Q_\phi(x \mid x', y) evaluated is the right way to stop reward optimisation from redefining the target which, as metnioned before, is highly problematic. As far as I can tell Prop 1 is correct as stated. The authors are also quite candid about the papers limitations since they report the sign test against tuned MCMC and show in Fig. 3 and Table 25 that the prior mean is already almost as good as the full pipeline for example. The experimental coverage is ok. They selected six families, tw to ten qubits with an exact two-qubit reference, and an enumerated chain where spectral gaps can be computed. The multimodal study is a sensible test too.

Critical weaknesses

a. The theory does not justify the main reconstruction results. Going over Appendix B one readsthat that the historical policies used features computed across chains in the same batch, so the frozen weights did not give each chain an independent frozen kernel. The 2-qubit and dense 4-qubit studies "have not been rerun". In addition, the text after Prop 1 says that such features "would require a joint-chain argument". Therefore in Fig. 2, the two-qubit energy statistic as well as Table 4 are descriptive outputs of a sampler that is not certified proper. However, the abstract and the second listed contribution present them as results of a target-preserving method. The corrected runs in Appendix O, which are covered by Prop 1, show a much smaller effect in practice.

b. The comparison with the flows is not apples to apples, and one realises that most of the aparent gain does not come from learning really. GRPO-QPS gets a physical-family prior and parameterisation that BuresTomFlow does not get. In general there is little to no evidence for GRPO.

c. When the authors work on "proper" learning and test fairly with the restricted class then the effect is null or negative.

d. The cost accounting is incomplete since ESS per likelihood call leaves out the cost of evaluating the policy network for both forward and reverse densities, as well as wall-clock time.

e. A smaller one is that the claim of exactness needs some more care. Likelihoods are computed with float32 matrix exponentials, with log-target errors of up to 4×10−34\times10^{-3} near 7,000 and while yes this is probably small, but its effect on the acceptance ratio should be bounded rather than assumed.

f. Novelty is limited. Eg learned MH proposals with exact correction are well established.

g. The presentation is actually quite hard to follow with 45 pages with many protocol revisions, and "historical", "exploratory" and corrected results sit side by side in the main text. It is honestly hard to tell which numbers are certified, which ones are not and overall this requires massive rework.

Questions for authors
  1. Please rerun the two-qubit, dense four-qubit and Fig. 2 comparisons with -fixed per-chain features- so that Prop 1 applies. Which of the main claims survive?

  2. How does a "by hand designed" mixture of the exact reflection and a Haario-adapted local kernel do on Table 1? How do parallel tempering and a flow-MCMC baseline do at equal likelihood and wall-clock budget?

  3. How many seeds and records underlie Table 1? Also please give ESS intervals, for example from batch means across independent chains, for Tables 1 and 2.

  4. Why did you choose GRPO specifically?

  5. Can you try bound the effect of float32 likelihood error on the acceptance probabilities in the 6-qubit runs?

Reviewer_fsZ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRPO-QPS, a framework for Bayesian quantum state tomography in which a learned proposal is placed inside a Metropolis–Hastings kernel, so the declared posterior (prior × Born likelihood) stays stationary after the policy is frozen.

Critical strengths
  • The core idea here is really solid. Splitting 'learning how to explore' from 'defining the target' using a full forward/reverse-density Metropolis correction is definitely the right move for scientific inference. It’s a clear step up from reward-tuned generators like Flow-GRPO, which don't offer any invariance guarantees. Proposition 1 holds up, and I appreciate that the authors are upfront about its limitations—specifically the frozen policy, state-only features, and the lack of claims around irreducibility or convergence rates.

  • The matched control, the prior-mean, MLE, shadow, RBM and neural-operator baselines (Fig. 3, Table 25) show that much of the headline gain over the flows is not due to RL.

  • The paper is clear about what each study can and cannot support (Tables 5, 6), the development-set status of the data, and the amortization cost of about 25.6M likelihood calls.

Critical weaknesses
  • The paper is hard to read. Roughly half the content is in appendices that the main text depends on, with many forward references.

  • The multimodal result is narrow, and the experimental design may favor the learned method. For example, the conventional baselines look under-tuned for this problem. There is no parallel tempering, no tempered or mode-jumping SMC, no HMC or NUTS in logit coordinates, and no mixture of local covariance-adapted moves with a reflection move.

  • The statistical support here is pretty thin overall. Almost all the results rely on development-set data using just three training repetitions or seeds. The confidence intervals are based on tt-distributions with only 2 degrees of freedom, and even the authors admit these findings shouldn't be taken as confirmatory. The paper’s strongest claims—like 'surpassing every conventional MCMC baseline tested'—depend on nine held-out comparisons that aren't actually independent, since they reuse three records across three training seeds. Additionally, the ESS numbers (like 102 vs. 28) come from single runs with no uncertainty reported, and the single-record results rely entirely on one record fixed by seed 1000.

  • Proposition 1 only establishes stationarity. Irreducibility is proven strictly for the 48-state graph; for the continuous mixture proposal, the authors don't provide any argument for ergodicity or mixing guarantees. In fact, the learned drift could theoretically lead to near-zero acceptance in certain regions, which seems to be what happened with the one failed seed. Calling the method 'target-preserving' is technically true, but only in a weak sense: it guarantees stationarity, but the chain might practically freeze up and fail to mix.

Questions for authors
  1. Since the matched non-learned control equals or beats GRPO-QPS on most structured families (Tables 15, 17), what is the claim in Section 4.1 attributable to GRPO? Would it be more accurate to retitle the result as “target-preserving MCMC from a flow initialization beats flows”? Also, can the flow baselines be given the same family-prior information (for instance projected onto the family, as in Table 18) so that the comparison isolates the sampling stage?

  2. For the multimodal benchmark, can you add (a) a reflection mixture combined with a Haario or covariance-adapted local move, (b) parallel tempering or tempered SMC, and (c) a gradient-based sampler (HMC/NUTS in logit coordinates, possibly with a reflection step)? Without these, it is hard to tell whether the advantage is due to learning or to the absence of a competent hybrid baseline. How much of the 102 vs 17 gap would a pilot-run covariance estimate close?

  3. The training objective uses reference moments from long prior-independence chains. How does this scale to settings where prior independence mixes poorly (the very regime motivating the method)?

  4. I was left wondering what causes the collapsed seed at four qubits, and how would a practitioner detect and avoid it at deployment without ground truth? Is there a safeguard (for example an acceptance floor or a mixture with a fixed kernel) built into the proposal, and does the guarantee on irreducibility hold for the learned Gaussian mixture on the bounded logit domain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.