acceptodds
Under review as a conference paper at ICLR 2027

Semi-Implicit Belief Approximation with Learned Transport and Reweighting for POMDPs

Abstract

Belief estimation in POMDPs requires approximating the posterior over latent states given a history of actions and observations. In many domains, however, the observations carry no information about the state for many consecutive timesteps, and the belief's errors accumulate. Keeping these errors small requires consistency in carrying the belief forward until the next informative observation and expressiveness in correcting it in a single update when that observation arrives. Existing belief representations assume that informative observations arrive regularly: they either rely on ground-truth states during training, or lose the hypothesis that the next informative observation supports. We propose Generative Belief Variational Inference (GBVI), built on the insight that any change in a belief decomposes into transport, which moves probability mass, and reweighting, which adjusts it in place. Semi-implicit variational inference splits the belief into components that form a density without a closed form, a memory-conditioned Hamiltonian flow transports them across timesteps, and a gated recurrent memory reweights them, with both operations learned from observations alone. On Symmetric Light-Dark, occluded Atari, and Area Search, GBVI outperforms recurrent, variational, particle-based, and state-supervised baselines, and on Area Search its belief error against the exact posterior stays low as the period without informative observations grows while the baselines' error rises.

Primary area:
reinforcement learning
Keywords:
Partially Observable Markov Decision Processes (POMDP), Belief Approximation, Semi-implicit Variational Inference, Hamiltonian Dynamics, Variational Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K1LsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GBVI for maintaining beliefs when an agent receives little useful information for many steps. It keeps several possible states, updates them over time, and adjusts their weights when new observations arrive. The model learns from observations and actions and is then used with PPO. Experiments cover Light-Dark, occluded Atari, and Area Search.

Critical strengths

S1. The paper addresses a practical problem: keeping track of several possibilities while a target is hidden and updating them when it reappears.

S2. Area Search compares the learned beliefs with an exact posterior. This provides useful evidence about belief quality alongside the task performance results.

S3. The reported improvements are promising. The experiments also examine the effects of weighting, component count, and different update methods.

Critical weaknesses

W1. The claim that hypotheses remain distinct is too strong. The proof considers components together with their momentum, while the algorithm discards and redraws momentum at each step. The retained components may still represent nearly the same hypothesis. The paper needs to clarify the guarantee and directly measure whether different hypotheses survive long observation gaps.

W2. The weighting network sees only the history, while the components are randomly initialized and updated. A given component index can therefore represent different hypotheses across runs. How these changing meanings stay aligned with the predicted weights remains unexplained.

W3. Jointly training the encoder and decoder to reconstruct the encoder’s own features creates a possible shortcut: both could output constant features and achieve low reconstruction error. A complete description of the training objective should specify the latent prior and any constraints that prevent this trivial solution.

Questions for authors

Q1. How much do the components overlap as observation gaps become longer?

Q2. How are component identities linked to the weights predicted from history?

Q3. What prevents feature collapse during pretraining, and how many independent training runs were used?

6 comments
AuthorsOpenReview ↗

Response to Reviewer K1Ls (Part 1): W1, Q1

We thank the reviewer for recognizing the importance of the problem, the new insight of the decomposition and the promise of the results. We address the questions below, and the changes are marked in blue in the revised paper.

W1 and Q1. The claim that hypotheses remain distinct is too strong, because the algorithm redraws the momentum at every step

With the independent momenta of Algorithm 1, the expected squared distance between two components has a lower bound, and this bound does not decrease with the length of the gap. Let ψi′\psi_i' and ψj′\psi_j' be the positions of two components after one transport step of Algorithm 1, which consists of L=2L = 2 Leapfrog steps of size ϵ\epsilon with independent momenta. Under Assumptions 8 and 10 with ϵ≤1\epsilon \le 1,

E ∥ψi′−ψj′∥2  ≥  4d ϵ2 (2−ϵ2LU)2,\mathbb{E}\,\|\psi_i' - \psi_j'\|^2 \;\ge\; 4d\,\epsilon^2\,(2 - \epsilon^2 L_U)^2 ,

where dd is the latent dimension and LUL_U is the Lipschitz constant of the gradient of the potential (Lemma 15). The bound holds at every step, even if two components start from the same position. Without the momenta, the only guarantee is the contraction bound of Lemma 14, which decays geometrically with the gap. A distance bound does not show that the components hold different hypotheses, so we also measure this directly.

Table E below compares GBVI with two variants on Area Search with gaps of 5 to 40 steps. The first is the same model with the momentum removed during the gap. The second, learned-GSF, keeps the transport of GBVI and multiplies a learned likelihood onto the previous weights, as the Gaussian-sum filter does (Table A in our response to Reviewer e2jq). The table reports the effective number of components, which is computed from their pairwise overlap and is 1 when all four coincide and 4 when none overlap. It also reports the share of the hypotheses of the exact posterior (the roads on which it has mass) that the belief retains on the last step of the gap.

Table E (mean ± standard error across training runs)

Gap length510203040
Effective number of components
GBVI1.49 ± 0.031.81 ± 0.012.27 ± 0.022.76 ± 0.063.16 ± 0.08
GBVI without momentum in the gap1.00 ± 0.001.00 ± 0.001.01 ± 0.011.01 ± 0.011.01 ± 0.01
learned-GSF1.47 ± 0.021.79 ± 0.022.29 ± 0.012.65 ± 0.093.14 ± 0.03
Share of hypotheses retained (higher is better)
GBVI0.99 ± 0.010.96 ± 0.010.92 ± 0.010.87 ± 0.010.82 ± 0.02
GBVI without momentum in the gap0.77 ± 0.010.64 ± 0.020.51 ± 0.010.39 ± 0.010.26 ± 0.01
learned-GSF0.94 ± 0.010.83 ± 0.010.75 ± 0.010.61 ± 0.030.47 ± 0.02

With the momentum the components move apart as the gap grows, and without it they merge and the hypotheses are lost. For GBVI, the effective number of components rises from 1.49 to 3.16 as the gap grows from 5 to 40 steps, and the belief retains 0.99 to 0.82 of the hypotheses. With the momentum removed, the effective number stays at 1.00 to 1.01, and the share retained falls from 0.77 to 0.26.

Separate components are necessary but not sufficient, because the weights can still lose the hypotheses. The components of learned-GSF are as far apart as those of GBVI (1.47 to 3.14), but the share it retains falls from 0.94 to 0.47, because its weights decay over the gap. The variant without momentum fails in the transport, and learned-GSF fails in the reweighting. GBVI keeps both intact, so its advantage grows with the gap.

We have replaced the earlier statement with this bound (Section 3.2, Appendices D and E), added its proof as Lemma 15 (Appendix E.2), and added Table E to Appendix F.9 with a summary in Section 4.3 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer K1Ls (Part 2): W2, Q2

We thank the reviewer for this question and continue our response from Part 1 below.

W2 and Q2. The weights are predicted from the history alone, so how do component identities stay aligned with them?

The link between the history and the individual components is made by the potential, which is read from the same memory state as the weights and takes the position of every component as its input. We agree that the weight head takes only the memory. However, the potential Uθ(ψ;h~t)U_\theta(\psi; \tilde h_t) is evaluated at the position of each component, and both readouts are trained together by Eq. (6), so the two operations are not independent.

To test whether the results rely on a match between indices and hypotheses, Table F gives the weight readout the position of each component in two ways, with the transport unchanged. The first reads the weights from the potential, πk∝exp⁡(−Uθ(ψ(k);h~t))\pi_k \propto \exp(-U_\theta(\psi^{(k)}; \tilde h_t)), which does not depend on the index. The second uses a separate network that takes the memory and the position.

Table F (mean ± standard error across training runs, lower is better)

Gap length510203040
Reweighting part of WFR² on the last step of the gap
GBVI0.04 ± 0.010.05 ± 0.010.05 ± 0.010.07 ± 0.020.09 ± 0.01
GBVI, weights from the potential0.05 ± 0.000.05 ± 0.010.05 ± 0.010.07 ± 0.010.08 ± 0.01
GBVI, weights from the positions0.04 ± 0.010.05 ± 0.010.06 ± 0.010.07 ± 0.010.09 ± 0.01
WFR² on the step the vehicle is seen again
GBVI0.07 ± 0.010.09 ± 0.010.08 ± 0.010.11 ± 0.010.13 ± 0.01
GBVI, weights from the potential0.07 ± 0.010.10 ± 0.020.09 ± 0.010.11 ± 0.010.12 ± 0.01
GBVI, weights from the positions0.08 ± 0.010.09 ± 0.010.09 ± 0.010.11 ± 0.010.12 ± 0.01

Giving the weight readout the positions of the components does not change the result. The three rows differ by at most 0.01 at every gap length, which is within one standard error. The results therefore do not rely on a fixed match between indices and hypotheses, and the reweighting can be replaced without changing the transport. This division of work follows the WFR geometry, in which any change of a distribution consists of transport, which moves mass, and a change of mass in place, which we call reweighting [1, 2]. We have added this explanation and Table F to Appendix F.10 of the revised paper.

[1] Chizat, Peyré, et al., An interpolating distance between optimal transport and Fisher–Rao metrics, Foundations of Computational Mathematics 2018.

[2] Yan, Wang, Rigollet, Learning Gaussian mixtures using the Wasserstein–Fisher–Rao gradient flow, arXiv 2023.

AuthorsOpenReview ↗

Response to Reviewer K1Ls (Part 3): W3, Q3, Q4

We thank the reviewer for these questions and conclude our response below.

W3 and Q3. What is the latent prior, and what prevents feature collapse?

The latent prior in Eq. (6) is p(zt)=N(0,I)p(z_t) = \mathcal N(0, I) and the decoder likelihood is p(xt∣zt)=N(xt; D(zt), I)p(x_t \mid z_t) = \mathcal N(x_t;\, D(z_t),\, I), so that p(ot,zt)=p(zt) p(xt∣zt)p(o_t, z_t) = p(z_t)\,p(x_t \mid z_t). We agree that a constant feature is a solution of this objective [1]. Existing work prevents this collapse by fixing the target, either with a stop-gradient on the target [1, 2] or with an encoder that is first trained to reconstruct the observation and then frozen [3]. Following these two approaches, we retrain GBVI on Area Search in two variants and compare them with the jointly trained version in Table G below.

Table G (WFR² on the last step of the gap, mean ± standard error across training runs, lower is better)

Gap length510203040
GBVI0.10 ± 0.010.11 ± 0.010.12 ± 0.010.14 ± 0.010.18 ± 0.02
GBVI, stop-gradient on the target0.09 ± 0.010.11 ± 0.020.12 ± 0.010.13 ± 0.010.17 ± 0.01
GBVI, encoder fixed0.10 ± 0.010.10 ± 0.010.12 ± 0.020.14 ± 0.010.18 ± 0.01

The two variants match GBVI at every gap length, so the results do not depend on the joint training. The three rows differ by at most 0.01, which is within one standard error. The belief update therefore gives the same result on three encoders that are trained in different ways. We have stated the prior and the decoder likelihood in Section 3.4 and added Table G to Appendix F.11 of the revised paper.

[1] Chen, He, Exploring simple Siamese representation learning, CVPR 2021.

[2] Ni, Eysenbach, et al., Bridging state and history representations: Understanding self-predictive RL, ICLR 2024.

[3] Ha, Schmidhuber, Recurrent world models facilitate policy evolution, NeurIPS 2018.

Q4. How many independent training runs were used?

Every entry of Table 1 is computed from 10 independent training runs, each of which repeats both the pretraining of the belief network and the training of the policy. Each run is evaluated on 10 episodes (100 on Light-Dark), and the ± is the standard error across the 10 per-run averages. We have stated the number of training runs in the caption of Table 1 of the revised paper.

Reviewer_e2jqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
3(+1)
Read review
Paper summary

The paper focuses on the problem of approximating and updating beliefs over latent states in POMDPs where informative observations arrive rarely. It reviews prior work and positions itself within it in detail. The proposed method combines existing approaches, such as Hamiltonian flows, semi-implicit variational inference into an algorithm termed Generative Belief Variational Inference that learns to represent beliefs solely from observations. Such a learned representation can be combined with existing RL algorithms and the combination with PPO achieves significantly better results than most baselines on several classical domains.

Critical strengths

The paper combines several advanced topics from different subfields in a novel way, resulting a single algorithm that addresses one of the key problems in POMDPs.

The proposed algorithm achieves excellent results compared to the baselines; however, several important questions about the reported numbers remain.

Post-Response: Thank you for a thorough response and the new experiments that have answered my questions satisfactorily.

Critical weaknesses

The paper is incredibly hard to follow and required multiple re-reads before I could (mostly) understand it. The authors have moved a substantial part of the paper into the appendix, yet large chunks of text are repeated (e.g., between the Introduction and Related Work, among other places).

Based on the language used in the abstract and the introduction, the authors present the idea of decomposing the belief into transport and reweighting (formalized in Proposition 1 and proved in Appendix E.2) as one of the paper’s contributions. However, this is the classical Gaussian-sum filter update of Alspach & Sorenson (1972), which is not cited. Additionally, the proposition and its proof normalize by the sum of the products of weights and evidence, whereas Algorithm 1 and Line 299 in Section 3.3 use a softmax, which is not discussed at all.

The reported results lead to several contradictions (described in my questions), and the discussion of these results overstates the numbers reported in Table 1 in a few places.

Questions for authors
  1. What do you mean that recurrent encoders only hold one hypothesis at a time?
  2. Line 216 states that each component in the Gaussian mixture carries one hypothesis about the hidden state. With K = 4, it seems that the belief can hold 4 hypotheses. What happens when a vehicle passes one or more intersections while outside the UAV sensor’s range?
  3. Table 1 states that the reported numbers are averages over 100 seeds. Considering that the Pong’s returns are bounded in [-21, 21], the maximum possible variance is attained when half of the points lie on each end, giving us Var = 441, StdDev = 21 and StdErr = 2.1. 11 of 14 baselines exceed the maximum possible StdErr even without looking at the reported means. Can you clarify what quantities the table reports?
  4. Regarding LightDark, the paper doesn’t describe or specify the environment in any way and additionally talks about two different metrics: Table 1 mentions success rate (which implies binary scores) while the conclusion says that "GBVI attains the highest return or ties for it on Symmetric LightDark". Can you please clarify what is being reported?
8 comments
AuthorsOpenReview ↗

Response to Reviewer e2jq (Part 1): W1, W2

We thank the reviewer for recognizing the importance of the problem, the novelty of the decomposition and the promise of the results. We address the questions below in terms of the setting with unknown models, the one-step correction and what Table 1 reports. The changes are marked in blue in the revised paper.

W1. Proposition 1 is the Gaussian-sum filter update, which is not cited

The Gaussian-sum filter (GSF) cannot be instantiated when the transition and observation models are unknown, which is the setting GBVI is built for. We agree that the update in Proposition 1 and the GSF update of [1] are the same Bayes update of a weighted mixture under known models. However, GSF needs both models. Its prediction step needs the transition function ff and the process-noise covariance QQ, and its correction step needs the observation function hh and the noise covariance RR. In contrast, our belief network sees only observations and actions. Therefore, neither step can be computed, and each has to be learned.

A learned transport must not contract the components (Corollary 2), so GBVI moves them with a fresh momentum at every step, and this randomness makes the belief a semi-implicit mixture rather than the Gaussian mixture of GSF. A learned deterministic map in place of ff pulls every component toward what the memory favours and merges the hypotheses during a gap, as a recurrent state does [2]. With the momenta, the components are random, so the belief is the average of the KK-Gaussian sum over the momenta and has no closed-form density.

The belief is nevertheless trained with the SIVI bound of [3], which Theorem 3 extends to components that persist across steps, so the transport and the reweighting are trained toward the Bayes update of Proposition 1 without the models and without a closed-form density. Eq. (6) computes the ELBO on the explicit mixture of each draw of the momenta and averages over the draws, which is the surrogate bound of [3]. Standard SIVI proves this bound for independent draws, and Theorem 3 shows that it still holds when the components depend on their previous positions and on each other through the shared memory. We have cited [1] and added this comparison to Sections 2 and 3.1 of the revised paper.

[1] Alspach, Sorenson, Nonlinear Bayesian estimation using Gaussian sum approximations, IEEE TAC 1972.

[2] Igl, Zintgraf, et al., Deep variational reinforcement learning for POMDPs, ICML 2018.

[3] Yin, Zhou, Semi-implicit variational inference, ICML 2018.

W2. Proposition 1 normalizes a product, whereas Algorithm 1 uses a softmax

Proposition 1 specifies what the reweighting must do, and the normalized product is its closed form when the observation model is known. However, GBVI reads the weights from the memory by softmax instead of multiplying a learned likelihood onto the previous weights, because a product cannot restore mass to a hypothesis whose weight has decayed during a gap (Corollary 2). The learned likelihoods are not exactly equal during the gap, so the product shrinks the weight of a hypothesis step by step, and a single finite ℓk\ell_k at the return cannot bring it back. In contrast, the softmax reads the weights from the memory that the gate has just rewritten, so it moves the mass in one update even when the weight has become very small.

Table A below compares GBVI with learned-GSF on Area Search with controlled gaps. This variant keeps GBVI's network and training and changes only the weight update to the product πk∝πk p(xt∣ψ(k))\pi_k \propto \pi_k\, p(x_t \mid \psi^{(k)}). With the product, the mass at return falls from 0.22 to 0.01 as the gap grows from 5 to 40 steps, whereas with the softmax readout it only falls from 0.29 to 0.22. The reweighting part of the error rises from 0.09 to 0.51 for learned-GSF and stays at or below 0.09 for GBVI. The transport part differs by at most 0.01 between the two, so the difference comes from the weight update alone. We have added the discussion of the softmax readout to Section 3.1 and Table A with its analysis to Appendix F.6 of the revised paper.

Table A (mean ± standard error across training runs)

L = 510203040
Mass at return (higher is better)
GBVI0.29 ± 0.040.27 ± 0.050.26 ± 0.030.24 ± 0.040.22 ± 0.02
learned-GSF0.22 ± 0.060.13 ± 0.080.08 ± 0.070.04 ± 0.010.01 ± 0.00
Reweighting part of WFR² (lower is better)
GBVI0.04 ± 0.010.05 ± 0.010.05 ± 0.010.07 ± 0.020.09 ± 0.01
learned-GSF0.09 ± 0.010.14 ± 0.030.24 ± 0.020.37 ± 0.060.51 ± 0.09
Transport part of WFR² (lower is better)
GBVI0.06 ± 0.010.06 ± 0.010.07 ± 0.010.07 ± 0.010.09 ± 0.01
learned-GSF0.06 ± 0.010.06 ± 0.010.07 ± 0.010.08 ± 0.010.08 ± 0.01
AuthorsOpenReview ↗

Response to Reviewer e2jq (Part 2): Q1, W3, Q3

We thank the reviewer for these questions and continue our response from Part 1 below.

Q1. What does it mean that recurrent encoders hold one hypothesis at a time?

A recurrent encoder stores the history in a single vector [1]. In principle this vector can encode several hypotheses, but in practice the belief read from it stays on only one of them. Hafner et al. [2] report that purely deterministic transitions prevent a latent dynamics model from capturing multiple futures. Igl et al. [3] observe that a recurrent network tends to remember features of the past instead of a belief state. This limitation motivated particle beliefs [1, 3] and stochastic latent states [2], which add randomness to the state. GBVI does the same: its transport draws a fresh momentum at every step, which keeps the components apart (see our response to W1).

To show what this means in our setting, consider Area Search. The multiple futures are the roads that the vehicle could be on after it passes intersections out of view, and holding one hypothesis means that the belief has mass on only one of them. Table B below reports the number of these roads on which the belief has mass, averaged over scenarios, for GBVI and the RG-LRU encoder on the scenarios of Table D (our response to Q2).

Table B (mean ± standard error across training runs, higher is better)

Intersections1234
GBVI1.84 ± 0.062.79 ± 0.043.81 ± 0.094.96 ± 0.12
RG-LRU0.80 ± 0.010.97 ± 0.020.86 ± 0.010.93 ± 0.01

On average over the scenarios, the belief of RG-LRU stays on about one road for any number of intersections (0.80 to 0.97), whereas the belief of GBVI covers 1.84 roads after one intersection and 4.96 after four. The advantage of GBVI therefore grows with the number of hypotheses that the gap creates. GBVI covers more than K=4K = 4 roads because one component can cover neighbouring roads (see our response to Q2). We have added this statement to Section 4.1 and Table B to Appendix F.8 of the revised paper.

[1] Ma, Karkus, et al., Particle filter recurrent neural networks, AAAI 2020.

[2] Hafner, Lillicrap, et al., Learning latent dynamics for planning from pixels, ICML 2019.

[3] Igl, Zintgraf, et al., Deep variational reinforcement learning for POMDPs, ICML 2018.

W3 and Q3. The standard errors reported for Pong exceed what 100 episodes allow, so what does Table 1 report?

The ± in Table 1 is the standard error across 10 seeds with 10 evaluation episodes each (100 on Light-Dark), so the largest possible standard error on Pong is 6.6 and not 2.1. As stated in Section 4.2, every method is evaluated over 100 episodes on shared seeds. These 100 seeded episodes are 10 episodes for each of 10 distinct seeds, where each seed is an independent training run. We first average the 10 episodes of each seed, and we then compute the standard error across the 10 averages. Each average lies in [−21,21][-21, 21], so the reviewer's bound becomes 21/10≈6.621/\sqrt{10} \approx 6.6, and every Pong entry in Table 1 is below it. To make this verifiable, Table C below reports both quantities for Pong.

Table C (return, higher is better)

GBVIRG-LRULRUHVAESIVAEESCORTWBU†Believer†
Pong (FI), over 100 episodes17.52 ± 0.2316.50 ± 1.1111.94 ± 1.4510.75 ± 1.5212.09 ± 0.9314.66 ± 1.3715.02 ± 0.8811.74 ± 1.44
Pong (FI), across 10 seeds (Table 1)17.52 ± 0.1416.50 ± 2.3611.94 ± 2.6810.75 ± 3.6212.09 ± 2.6514.66 ± 2.6915.02 ± 1.1411.74 ± 2.94
Pong (FL), over 100 episodes17.08 ± 0.1315.26 ± 0.9512.27 ± 1.0912.00 ± 1.1311.92 ± 1.2315.89 ± 0.9915.94 ± 1.2214.93 ± 1.30
Pong (FL), across 10 seeds (Table 1)17.08 ± 0.1215.26 ± 2.4412.27 ± 3.2212.00 ± 3.0511.92 ± 2.9215.89 ± 1.1315.94 ± 2.1114.93 ± 1.97

The standard error over the 100 pooled episodes is below 2.1 for every method, which agrees with the reviewer's calculation, whereas the standard error across the 10 seeds is the value in Table 1. We have revised the caption of Table 1 and Appendix G.2 to clarify this computation, and added Table C to Appendix F.13 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer e2jq (Part 3): Q2

We thank the reviewer for this question and continue our response below.

Q2. With K = 4, what happens when the vehicle passes one or more intersections while out of the sensor's range?

When the roads outnumber the components, the belief either becomes coarser or loses a hypothesis, depending on KK, and the transport–reweighting decomposition identifies which of the two occurs. A component is a Gaussian whose centre and spread are both updated by the transport (Eq. 1–2), so it can widen as well as move. One component can therefore cover several neighbouring roads, and KK of them can hold mass on more than KK roads. Two outcomes are then possible. If the components still cover all the roads, every road has mass nearby, and matching the exact posterior only requires moving this mass a short distance. This error appears in the transport part of WFR². If KK is too small, some roads are left with no mass at all, and the missing mass has to be created. This error appears in the reweighting part (Corollary 2). The part that grows with the number of intersections therefore tells us which outcome has occurred.

To show this, we generated scenarios in which the vehicle passes one to four intersections during a gap of fixed length. Table D below reports the two parts of WFR² for K∈{1,2,4,16}K \in \{1, 2, 4, 16\}.

Table D (mean ± standard error across training runs, lower is better)

Intersections1234
Transport part of WFR²
K=1K = 10.08 ± 0.010.09 ± 0.010.10 ± 0.010.09 ± 0.02
K=2K = 20.08 ± 0.010.08 ± 0.010.11 ± 0.020.10 ± 0.01
GBVI (K=4K = 4)0.05 ± 0.010.07 ± 0.010.13 ± 0.020.17 ± 0.02
K=16K = 160.06 ± 0.010.08 ± 0.020.15 ± 0.020.17 ± 0.01
Reweighting part of WFR²
K=1K = 10.09 ± 0.010.17 ± 0.020.29 ± 0.040.47 ± 0.07
K=2K = 20.11 ± 0.020.21 ± 0.010.27 ± 0.020.39 ± 0.04
GBVI (K=4K = 4)0.07 ± 0.010.07 ± 0.020.09 ± 0.020.14 ± 0.02
K=16K = 160.06 ± 0.010.07 ± 0.010.10 ± 0.010.17 ± 0.02

With K=1K = 1 and K=2K = 2 the belief loses roads as the intersections increase, whereas the belief of GBVI (K=4K = 4) becomes coarser instead. For K=1K = 1 and K=2K = 2, the reweighting part rises from 0.09 to 0.47 and from 0.11 to 0.39, while the transport part stays between 0.08 and 0.11. For GBVI, the reweighting part stays at or below 0.14, and the error grows mainly in the transport part, from 0.05 to 0.17. K=16K = 16 is within 0.03 of K=4K = 4 in both parts, so increasing KK beyond the value used in the paper brings no further gain in these scenarios. This is the practical role of the decomposition: it shows which half of the belief update a representation fails on. We have revised the sentence of Line 216 in Section 3.1 accordingly, summarized this result in Section 4.3, and added Table D to Appendix F.7 of the revised paper.

Reviewer_Eq2rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce Generative Belief Variational Inference (GBVI), a belief representation built on a transport + reweighting decomposition designed to overcome error accumulation due to sparse informative observations in POMDPs over a learned latent state. More concretely, GBVI extends SIVI to partially observable models by modeling belief as the marginal of a generative process over K components, transporting those components with a memory-conditioned Hamiltonian flow, and reweighting with an RG-LRU memory. Transport and reweighting are learned via observations without access to privileged information in training. GBVI is coupled with a policy gradient method (PPO) and evaluated on standard + modified POMDP benchmarks to include Light-Dark, Atari, and Area Search, where it is shown to outperform several baselines. Area Search has an exact posterior, and provides an opportunity to decompose belief error into transport and reweight portions; here, GBVI maintains a low level of error accumulation even as informative observation sparsity increases.

Critical strengths

1.) The paper is comprehensive in its presentation (approach: belief, transport, reweighting, and training/objective), supported by theoretical analysis - however straightforward (e.g. the Bayes update formalization as a transport + reweighting step), and substantive in its description, with an appendix that carefully documents environments, hyperparameters, and ablations. I generally find the manuscript well written. Assumptions are clearly outlined in the appendix. 2.) Sparse informative evidence distorting belief in learned partially observable decision processes is a known issue and the architecture proposed is well-motivated by the transport + reweighting decomposition. Experimental results are strong, with seven baselines spanning recurrent, variational, particle, and state-supervised approaches. The belief error on area search results are especially compelling.

Critical weaknesses
  1. The explicit definition of p(ot,zt)p(o_t, z_t), used in Equation 6 and Theorem 3, is not clear to me from the main manuscript. Is this a static VAE-style joint probability, as in the standard SIVI setup (which would preclude temporal coupling, preventing transport from tracking the state through gaps in informative evidence), or something more akin to a sequential joint probability with a learned transition prior?
  2. Significant mention is made of the "no closed-form density" motivation for SIVI, which underlies GBVI. But if I understand correctly, conditional on a sampled set of momenta, the belief is a K-component Gaussian mixture with an explicit density, as in fixed prior to training much like the other baselines mentioned?
Questions for authors
  1. The paper argues that GBVI is modular and integrates with existing RL frameworks. Why PPO only, and does integration with a model-based algorithm affect what trains the transport?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Eq2r (Part 1): W1, W2

We thank the reviewer for the positive assessment of the problem, the decomposition and the results, and for finding the presentation comprehensive. We address the questions below, and the changes are marked in blue in the revised paper.

W1. Is p(ot,zt)p(o_t, z_t) a static joint or a sequential joint with a transition prior?

The joint is the static per-step joint of standard SIVI [1], with no transition prior, and the temporal coupling is in the belief instead. Specifically, p(ot,zt)=p(zt) p(xt∣zt)p(o_t, z_t) = p(z_t)\,p(x_t \mid z_t) with p(zt)=N(0,I)p(z_t) = \mathcal N(0, I) and p(xt∣zt)=N(xt; D(zt), I)p(x_t \mid z_t) = \mathcal N(x_t;\, D(z_t),\, I), where xtx_t is the encoder feature of oto_t. Eq. (6) uses this joint only to score the belief. The belief at step tt is formed by moving the components from their positions at step t−1t-1 and reading the weights from the memory, so it depends on all earlier observations although the joint contains only the current one. Accordingly, Theorem 3 now states the bound as L(t)≤log⁡p(ot)\mathcal L^{(t)} \le \log p(o_t). In contrast, standard SIVI and our SIVAE baseline redraw ψ\psi at every step, so their components do not persist from one step to the next.

We do not use a transition prior because, during a gap, a learned transition model [2, 3] has to predict many steps without correction, and its errors accumulate with the number of steps [4, 5]. In contrast, GBVI trains the transport over the whole gap. Each component moves only a bounded distance per step. To explain the observation at the end of a gap, the components therefore have to move toward it during the gap. Consistent with this, the WFR² of GBVI to the exact posterior rises only from 0.10 to 0.18 as the gap grows from 5 to 40 steps (Table G in our response to Reviewer K1Ls). We have stated the definition of p(ot,zt)p(o_t, z_t) with Eq. (6) and clarified Theorem 3 in Section 3.4 of the revised paper.

W2. Why is "no closed-form density" the motivation for SIVI?

The momenta are needed to keep the components apart, and with them the belief has no closed-form density, which is why we use SIVI. The absence of a closed form is therefore a consequence of the momenta and not a goal. We agree that the belief is an explicit mixture of KK Gaussians once the momenta are fixed. However, the belief of Eq. (3) is the average of this mixture over the momenta, and this average has no closed-form density. Eq. (6) computes the ELBO on the explicit mixture of each draw and averages over the draws, which is the SIVI bound [1], and Theorem 3 shows that it still holds when the components persist across steps. Only the number of components and their Gaussian shape are fixed before training, whereas their positions and weights are learned.

The components merge and the hypotheses are lost without the momenta. When the momenta are removed during the gap, the share of the hypotheses of the exact posterior that the belief retains after 40 steps falls to 0.26, against 0.82 for GBVI (Table E in our response to Reviewer K1Ls, and Table 3 of the revised paper). We have clarified this order of reasoning in the Introduction and Section 3.1 of the revised paper.

[1] Yin, Zhou, Semi-implicit variational inference, ICML 2018.

[2] Igl, Zintgraf, et al., Deep variational reinforcement learning for POMDPs, ICML 2018.

[3] Hafner, Lillicrap, et al., Learning latent dynamics for planning from pixels, ICML 2019.

[4] Janner, Fu, et al., When to trust your model: Model-based policy optimization, NeurIPS 2019.

[5] Lambert, Pister, Calandra, Investigating compounding prediction errors in learned dynamics models, arXiv 2022.

AuthorsOpenReview ↗

Response to Reviewer Eq2r (Part 2): Q1

We thank the reviewer for this question and conclude our response below.

Q1. Why PPO only, and would a model-based algorithm affect what trains the transport?

The belief is trained in Stage 1 by Eq. (6) from observations and actions and is then frozen, so the loss of the Stage 2 learner never reaches the transport or the reweighting. Replacing the learner, including by a model-based one, therefore does not change what trains the transport. Running a learner on a separately trained representation is common for both model-free [1] and model-based [2] algorithms. We used PPO because it handles both the continuous actions of Light-Dark and the discrete actions of Atari with one implementation.

To test whether the advantage depends on PPO, we kept the frozen beliefs of GBVI and of the two recurrent baselines and trained the policy with a second learner, SAC on Light-Dark and DQN on two Atari games. Table H uses a smaller training budget than Table 1, so its numbers should not be compared with Table 1.

Table H (return, mean ± standard error, higher is better)

GBVIRG-LRULRU
Light-Dark, PPO0.85 ± 0.010.80 ± 0.020.78 ± 0.02
Light-Dark, SAC0.81 ± 0.010.74 ± 0.010.70 ± 0.02
Bowling (FI), PPO31.94 ± 0.3928.17 ± 0.4227.43 ± 0.23
Bowling (FI), DQN33.28 ± 0.3625.24 ± 0.9621.84 ± 0.44
BeamRider (FI), PPO1194.87 ± 23.79980.73 ± 43.87952.93 ± 33.06
BeamRider (FI), DQN1849.20 ± 25.781593.20 ± 50.741187.48 ± 21.29

The ranking is the same in all six rows, with GBVI the highest, and the gap to RG-LRU does not shrink when PPO is replaced (0.05 and 0.07 on Light-Dark, 3.77 and 8.04 on Bowling, 214 and 256 on BeamRider). The advantage therefore comes from the belief and not from a particular learner. We have added Table H to Section 4.2 (Table 4) and to Appendix F.12 of the revised paper.

[1] Stooke, Lee, et al., Decoupling representation learning from reinforcement learning, ICML 2021.

[2] Zhou, Pan, et al., DINO-WM: World models on pre-trained visual features enable zero-shot planning, arXiv 2024.

AuthorsOpenReview ↗

Follow-up to Reviewer Eq2r

Dear Reviewer Eq2r,

We sincerely thank you for your positive and constructive review, which has significantly contributed to improving our paper. As the discussion period closes on November 18, we would like to check whether our responses have answered your questions. We are happy to clarify anything further, and if they have, we would be grateful if you could consider reflecting this in your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revision: We Look Forward to the Discussion

We sincerely thank the Area Chair and all reviewers for their time and valuable assessments. This paper studies belief estimation through long periods without informative observations and proposes GBVI with a novel insight: the belief update decomposes into transport, which moves the belief through such a period, and reweighting, which corrects it when an informative observation arrives. GBVI shows high belief quality on UAV Area Search and promising returns on Light-Dark and occluded Atari.

In their initial reviews, the reviewers highlighted:

  1. All three reviewers recognized that the paper addresses a key practical problem in POMDPs.
  2. The reviewers valued the novelty of the approach (e2jq), its motivation by the transport–reweighting decomposition (K1Ls, Eq2r), and its compatibility with existing RL algorithms (e2jq, Eq2r).
  3. All three reviewers found the results promising, and K1Ls and Eq2r highlighted the belief error against the exact posterior on Area Search as evidence of belief quality.

To address the questions raised, we further support these highlights with added experiments, theory and related work:

  • Novelty of the approach: we show that, unlike the Gaussian-sum filter, GBVI learns both operations without the transition and observation models, and its softmax readout keeps the mass over a long gap (Table A; e2jq W1, W2).
  • Motivation by the decomposition: we show that it identifies which half of the belief update fails, by varying the intersections passed and the number of components (Tables B, D; e2jq Q1, Q2).
  • Transport: we show that the momenta keep the components apart, which is also why the belief is trained with SIVI (Lemma 15, Table E; K1Ls W1, Q1; Eq2r W1, W2).
  • Reweighting: we show that the results do not rely on a match between indices and hypotheses, and that the reweighting can be replaced without changing the transport (Table F; K1Ls W2, Q2).
  • Compatibility with RL algorithms: we show that the ranking stays the same when PPO is replaced by SAC or DQN (Table H; Eq2r Q1).
  • Promising results: we show that they do not depend on the joint training of the encoder (Table G; K1Ls W3, Q3), and we clarify what Table 1 reports and describe Light-Dark (e2jq W3, Q3, Q4; K1Ls Q4).

Full results are in the individual responses, and all changes are marked in blue in the revised paper. We thank the Area Chair and all reviewers again, look forward to the discussion, and would be grateful if the reviewers could consider raising their ratings in light of these highlights and the added evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.