- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GBVI for maintaining beliefs when an agent receives little useful information for many steps. It keeps several possible states, updates them over time, and adjusts their weights when new observations arrive. The model learns from observations and actions and is then used with PPO. Experiments cover Light-Dark, occluded Atari, and Area Search.
Critical strengths
S1. The paper addresses a practical problem: keeping track of several possibilities while a target is hidden and updating them when it reappears.
S2. Area Search compares the learned beliefs with an exact posterior. This provides useful evidence about belief quality alongside the task performance results.
S3. The reported improvements are promising. The experiments also examine the effects of weighting, component count, and different update methods.
Critical weaknesses
W1. The claim that hypotheses remain distinct is too strong. The proof considers components together with their momentum, while the algorithm discards and redraws momentum at each step. The retained components may still represent nearly the same hypothesis. The paper needs to clarify the guarantee and directly measure whether different hypotheses survive long observation gaps.
W2. The weighting network sees only the history, while the components are randomly initialized and updated. A given component index can therefore represent different hypotheses across runs. How these changing meanings stay aligned with the predicted weights remains unexplained.
W3. Jointly training the encoder and decoder to reconstruct the encoder’s own features creates a possible shortcut: both could output constant features and achieve low reconstruction error. A complete description of the training objective should specify the latent prior and any constraints that prevent this trivial solution.
Questions for authors
Q1. How much do the components overlap as observation gaps become longer?
Q2. How are component identities linked to the weights predicted from history?
Q3. What prevents feature collapse during pretraining, and how many independent training runs were used?
6 comments
Response to Reviewer K1Ls (Part 2): W2, Q2
We thank the reviewer for this question and continue our response from Part 1 below.
W2 and Q2. The weights are predicted from the history alone, so how do component identities stay aligned with them?
The link between the history and the individual components is made by the potential, which is read from the same memory state as the weights and takes the position of every component as its input. We agree that the weight head takes only the memory. However, the potential is evaluated at the position of each component, and both readouts are trained together by Eq. (6), so the two operations are not independent.
To test whether the results rely on a match between indices and hypotheses, Table F gives the weight readout the position of each component in two ways, with the transport unchanged. The first reads the weights from the potential, , which does not depend on the index. The second uses a separate network that takes the memory and the position.
Table F (mean ± standard error across training runs, lower is better)
| Gap length | 5 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|
| Reweighting part of WFR² on the last step of the gap | |||||
| GBVI | 0.04 ± 0.01 | 0.05 ± 0.01 | 0.05 ± 0.01 | 0.07 ± 0.02 | 0.09 ± 0.01 |
| GBVI, weights from the potential | 0.05 ± 0.00 | 0.05 ± 0.01 | 0.05 ± 0.01 | 0.07 ± 0.01 | 0.08 ± 0.01 |
| GBVI, weights from the positions | 0.04 ± 0.01 | 0.05 ± 0.01 | 0.06 ± 0.01 | 0.07 ± 0.01 | 0.09 ± 0.01 |
| WFR² on the step the vehicle is seen again | |||||
| GBVI | 0.07 ± 0.01 | 0.09 ± 0.01 | 0.08 ± 0.01 | 0.11 ± 0.01 | 0.13 ± 0.01 |
| GBVI, weights from the potential | 0.07 ± 0.01 | 0.10 ± 0.02 | 0.09 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 |
| GBVI, weights from the positions | 0.08 ± 0.01 | 0.09 ± 0.01 | 0.09 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 |
Giving the weight readout the positions of the components does not change the result. The three rows differ by at most 0.01 at every gap length, which is within one standard error. The results therefore do not rely on a fixed match between indices and hypotheses, and the reweighting can be replaced without changing the transport. This division of work follows the WFR geometry, in which any change of a distribution consists of transport, which moves mass, and a change of mass in place, which we call reweighting [1, 2]. We have added this explanation and Table F to Appendix F.10 of the revised paper.
[1] Chizat, Peyré, et al., An interpolating distance between optimal transport and Fisher–Rao metrics, Foundations of Computational Mathematics 2018.
[2] Yan, Wang, Rigollet, Learning Gaussian mixtures using the Wasserstein–Fisher–Rao gradient flow, arXiv 2023.
Response to Reviewer K1Ls (Part 3): W3, Q3, Q4
We thank the reviewer for these questions and conclude our response below.
W3 and Q3. What is the latent prior, and what prevents feature collapse?
The latent prior in Eq. (6) is and the decoder likelihood is , so that . We agree that a constant feature is a solution of this objective [1]. Existing work prevents this collapse by fixing the target, either with a stop-gradient on the target [1, 2] or with an encoder that is first trained to reconstruct the observation and then frozen [3]. Following these two approaches, we retrain GBVI on Area Search in two variants and compare them with the jointly trained version in Table G below.
Table G (WFR² on the last step of the gap, mean ± standard error across training runs, lower is better)
| Gap length | 5 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|
| GBVI | 0.10 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 | 0.14 ± 0.01 | 0.18 ± 0.02 |
| GBVI, stop-gradient on the target | 0.09 ± 0.01 | 0.11 ± 0.02 | 0.12 ± 0.01 | 0.13 ± 0.01 | 0.17 ± 0.01 |
| GBVI, encoder fixed | 0.10 ± 0.01 | 0.10 ± 0.01 | 0.12 ± 0.02 | 0.14 ± 0.01 | 0.18 ± 0.01 |
The two variants match GBVI at every gap length, so the results do not depend on the joint training. The three rows differ by at most 0.01, which is within one standard error. The belief update therefore gives the same result on three encoders that are trained in different ways. We have stated the prior and the decoder likelihood in Section 3.4 and added Table G to Appendix F.11 of the revised paper.
[1] Chen, He, Exploring simple Siamese representation learning, CVPR 2021.
[2] Ni, Eysenbach, et al., Bridging state and history representations: Understanding self-predictive RL, ICLR 2024.
[3] Ha, Schmidhuber, Recurrent world models facilitate policy evolution, NeurIPS 2018.
Q4. How many independent training runs were used?
Every entry of Table 1 is computed from 10 independent training runs, each of which repeats both the pretraining of the belief network and the training of the policy. Each run is evaluated on 10 episodes (100 on Light-Dark), and the ± is the standard error across the 10 per-run averages. We have stated the number of training runs in the caption of Table 1 of the revised paper.
Response to Reviewer K1Ls (Part 1): W1, Q1
We thank the reviewer for recognizing the importance of the problem, the new insight of the decomposition and the promise of the results. We address the questions below, and the changes are marked in blue in the revised paper.
W1 and Q1. The claim that hypotheses remain distinct is too strong, because the algorithm redraws the momentum at every step
With the independent momenta of Algorithm 1, the expected squared distance between two components has a lower bound, and this bound does not decrease with the length of the gap. Let ψi′ and ψj′ be the positions of two components after one transport step of Algorithm 1, which consists of L=2 Leapfrog steps of size ϵ with independent momenta. Under Assumptions 8 and 10 with ϵ≤1,
E∥ψi′−ψj′∥2≥4dϵ2(2−ϵ2LU)2,
where d is the latent dimension and LU is the Lipschitz constant of the gradient of the potential (Lemma 15). The bound holds at every step, even if two components start from the same position. Without the momenta, the only guarantee is the contraction bound of Lemma 14, which decays geometrically with the gap. A distance bound does not show that the components hold different hypotheses, so we also measure this directly.
Table E below compares GBVI with two variants on Area Search with gaps of 5 to 40 steps. The first is the same model with the momentum removed during the gap. The second, learned-GSF, keeps the transport of GBVI and multiplies a learned likelihood onto the previous weights, as the Gaussian-sum filter does (Table A in our response to Reviewer e2jq). The table reports the effective number of components, which is computed from their pairwise overlap and is 1 when all four coincide and 4 when none overlap. It also reports the share of the hypotheses of the exact posterior (the roads on which it has mass) that the belief retains on the last step of the gap.
Table E (mean ± standard error across training runs)
With the momentum the components move apart as the gap grows, and without it they merge and the hypotheses are lost. For GBVI, the effective number of components rises from 1.49 to 3.16 as the gap grows from 5 to 40 steps, and the belief retains 0.99 to 0.82 of the hypotheses. With the momentum removed, the effective number stays at 1.00 to 1.01, and the share retained falls from 0.77 to 0.26.
Separate components are necessary but not sufficient, because the weights can still lose the hypotheses. The components of learned-GSF are as far apart as those of GBVI (1.47 to 3.14), but the share it retains falls from 0.94 to 0.47, because its weights decay over the gap. The variant without momentum fails in the transport, and learned-GSF fails in the reweighting. GBVI keeps both intact, so its advantage grows with the gap.
We have replaced the earlier statement with this bound (Section 3.2, Appendices D and E), added its proof as Lemma 15 (Appendix E.2), and added Table E to Appendix F.9 with a summary in Section 4.3 of the revised paper.