- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes BETS (Belief Embedding Tree Search), a Monte Carlo planner for POMDPs that represents the belief at every search-tree node with a generative distribution embedding (GDE) instead of a particle set. Expanding a leaf is one Neural Bayesian Filtering (NBF) update, which regenerates particles from the parent node’s embedding, propagates them through the known dynamics, reweights them by the observation likelihood, and re-embeds the weighted set — so the impoverishment that accumulates under repeated resampling in particle filters does not accumulate across updates. Prior methods such as BetaZero also guide the search with learned policy and value networks, but condition them on hand-chosen summary statistics of a particle filter (per-variable mean and standard deviation), which cannot represent dependence between state variables or multimodal marginals. BETS conditions its networks directly on the embedding, and extends GDEs with a variable-coupling stream that pools pairwise co-occurrence statistics explicitly. The belief model and both networks are trained jointly on self-collected episodes. On six configurations of RockSample, LightDark, and PocMan, BETS is competitive or leading everywhere, with large margins on the configurations with the highest-dimensional beliefs.
Critical strengths
-
Well-motivated, elegant design. The motivation is laid out clearly: (1) tracking beliefs with particle filters suffers from long-term particle impoverishment, and (2) summarizing the particle set with per-variable statistics (as BetaZero does) discards the dependence between state variables and the shape of the marginals. BETS keeps BetaZero’s overall loop but replaces the particle set with a single object — the NBF embedding — that satisfies the three requirements the paper states: updatable from the parent node through the known dynamics, sampleable for simulation, and learned so the networks can condition on it.
-
The value-prediction experiment (Table 2b) isolates the representational claim from search effects: with identical value-network architectures and data, embeddings predict returns more accurately than the moments of a particle filter with 16x as many particles. This turns “higher returns” into direct evidence for the paper’s stated mechanism.
Critical weaknesses
-
The variable-coupling stream is underspecified, and its contribution cannot be verified as written. The mechanism in Section 3.3 appears mostly in the Figure 2 caption; Appendix C.1 covers only the generator, and the paper contains no encoder equations. Three components are left undefined: (a) the “joint feature of each pair of variables” averaged over samples is never defined; (b) the per-variable “variable summaries” appear only in the caption, so the block diagram cannot be reconstructed from the main text; (c) the attention-bias mechanism is unspecified (how is the pair feature projected into a bias?). Finally, the motivating b_excl/b_co example is not tested empirically. A small synthetic experiment, using the two-cell example or k-bit parity beliefs, could demonstrate that the pooled encoder fails to capture the joint distribution while VC recovers it.
-
No learning-based baseline outside the known-model planning family. The policy network alone outperforms every baseline with search on RockSample(20,20) and both PocMan mazes, suggesting that, once it learns a sufficient belief statistic, acting on these benchmarks is largely reactive. The paper therefore cannot rule out that a recurrent model-free agent (e.g., recurrent PPO or R2D2), trained at a matched budget of simulated environment interactions — optionally with an asymmetric critic to match BETS’s privileged access to the realized hidden state during training — would be competitive without any explicit belief tracking.
Questions for authors
First, I have the following suggestions for improving the clarity of the paper:
-
Figure 1 shows a single edge for each action at both the root and internal nodes. However, my understanding is that each action node also branches over the sampled observations. Please revise the figure or its caption to clarify this point, as the current depiction was misleading on a first read.
-
Including pseudocode for the training loop would improve the clarity of Section 4.
-
I encourage the authors to expand Section 3.3 to provide a more detailed explanation of the proposed variable-coupling embeddings.
-
A small empirical evaluation based on the b_excl/b_co example would provide useful empirical support for the main argument.
Regarding questions, my main question is how the proposed methods compare with model-free approaches.
We thank the reviewer for their review and useful suggestions which led to substantial improvements.
Based on your suggestions, we have added additional details to improve clarity of the paper. We have revised Figure 1 and its caption to include observation nodes. Appendix A contains pseudocode detailing the training procedure. Section 3.3, Appendix D.1 and Table 3 now contain details about the (fixed) variable-coupling stream, including the three missing pieces mentioned in the review. We apologize for not including them in the initial version.
We have also evaluated variable-coupling embeddings in a synthetic task based on the example from Section 2.2 which led us to an edge-case that the original VC formulation didn't cover. Please see the experiment, the fix and the results in the general response. The revised submission contains description of the fixed version.
Regarding the requested model-free baseline, we implemented Recurrent PPO and ran it on the smaller configuration of each domain. A GRU actor-critic replaces both the belief model and the search. At every step the policy receives the observation, its previous action and the same features of the observed part of the state that the BETS policy and value networks receive; the critic uses the two-hot value parameterization of BETS. We ran it with a standard critic and, as suggested, with an asymmetric critic that also sees the full simulator state during training. Evaluation follows the same setup as in the paper: three training seeds, 300 episodes at each of three evaluation seeds, greedy actions, mean discounted return with the SEM across training seeds.
Recurrent PPO is above BETS on LightDark(5), where the belief is one-dimensional. On PocMan(M) it reaches about 60 to 65% of the return of BETS with search (above BetaZero's 40.87), its training curves are flat after 5×107 steps, and the asymmetric critic adds about 5. On RockSample(15, 15), under-tuned PPO stays on a trivial policy at all three seeds (4.35): it checks and samples the rock next to the start and then idles until the horizon. With a constant learning rate, an entropy bonus annealed from 0.2 to 0.01 and four times the budget of the other two domains, it reaches the numbers in the table: about three quarters of the return of BETS with the standard critic, and less with the asymmetric one.
Each PPO run trains on 1.3×108 environment steps (5.2×108 on RockSample), roughly 100 times the real environment steps of a BETS run. The general response contains a table listing estimated training costs of BETS, BetaZero and Recurrent PPO. We commit to including Recurrent PPO as a baseline in all six configurations in the final version of the paper.