- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses how to effectively encode the outputs of particle filters under partial observability. The authors consider several encoding approaches, including statistical features, cumulant-generating functions, and neural networks, while also investigating the effects of pretraining on the encoders.
Critical strengths
Major Strengths
Particle filters play an important role in reinforcement learning under partial observability, and the research direction explored by the authors is helpful for addressing this problem.
Minor Strengths
The paper conducts experiments with five random seeds, increasing the reliability of the results.
Critical weaknesses
- The encoding methods employed in this paper have already been studied in previous research, raising concerns about the novelty of the paper. In particular, architectures such as Set Transformer have already been applied in prior work [1]. Furthermore, the experimental evaluation is limited, and broader comparisons with existing methods and more control tasks are needed to strengthen the paper's contributions.
[1] Abdulsamad, Hany, Sahel Mohammad Iqbal, and Simo Sarkka. "Sequential Monte Carlo for Policy Optimization in Continuous POMDPs." Advances in Neural Information Processing Systems 38 (2026): 41241–41269.
- The discussion of pretraining is insufficient. The authors show that neural network-based encoders, such as Set Transformer, can achieve better performance after pretraining, which suggests an issue related to sample efficiency. However, representation learning for reinforcement learning has already been extensively studied [2, 3]. The paper only considers offline pretraining, although offline data can be difficult to obtain.
[2] Schwarzer, Max, et al. "Data-Efficient Reinforcement Learning with Self-Predictive Representations." arXiv preprint arXiv:2007.05929 (2020).
[3] Zheng, Ruijie, et al. "TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning." Advances in Neural Information Processing Systems 36 (2023): 48203–48225.
Questions for authors
-
Can the authors provide a more detailed analysis of the mechanisms behind the performance differences among various encoders, including the effects of parameter scales, network depths, and other architectural factors?
-
Is the sample efficiency issue associated with neural network-based encoders related to the dimensionality of observations? Can the authors provide further analysis of this relationship?
-
Can the authors investigate whether incorporating online representation learning could improve task performance, rather than relying solely on offline pretraining, as explored in related work [2, 3]?
-
Can the authors provide additional evaluations on a broader range of control tasks, such as environments similar to those in [1], where some observable state information is masked to introduce partial observability, along with comparisons against more existing methods?
2 comments
Part 2 of 2
W2. The discussion of pretraining is insufficient, representation learning for RL is well studied, and only offline pretraining is considered.
We agree that this literature was missing. Section 2 now places our pretraining within it. In RL from images, representation learning can be decoupled from policy learning by pretraining the encoder and using it frozen (ATC, Stooke et al.), which is the route we follow for particle beliefs. Alternatively, self-supervised losses can be trained jointly with the RL objective (SPR, TACO), which we do not study. The Conclusion lists jointly trained objectives as future work.
We add two clarifications on the points raised.
- On offline data being hard to obtain, reconstruction pretraining needs only beliefs, with no rewards and no labels. For Faint and Diffuse Beacon these are drawn from the task's initial-layout distribution without any rollouts. For the other tasks they come from scripted rollouts (Appendix H). Task-specific pretraining does need labels read from simulator state, and Appendix H now says so.
- On sample efficiency, we agree with this reading for end-to-end training. Within our RL budget, the PPO objective alone does not train the Set Transformer to a useful encoding on several tasks, while a suitable pretrained start does. We did not test whether a longer RL budget would close the gap. One qualification is that a head start alone is not enough. On Faint and Diffuse Beacon, reconstruction pretraining leaves the Set Transformer at 13% success while task-specific pretraining brings it to 88% and 91%. What the pretraining objective teaches the encoder matters, and not only that the encoder is pretrained.
Q1. Mechanisms behind the differences, including parameter scale, depth, and architecture.
Appendix F now reports the size of every representation. The three set networks are matched in size (107,840 parameters for Deep Sets and PointNet, 109,640 for the Set Transformer on the 2-D tasks) and in output width (64). The differences between them are therefore not explained by parameter count. The architectural difference we point to in Section 6 is that the Set Transformer models interactions between particles before pooling, while Deep Sets and PointNet encode each particle independently. This matches where the Set Transformer leads after pretraining (tasks that require comparing particle clusters). We did not vary depth or width, and we do not claim an analysis of those factors.
Q2. Is the sample-efficiency issue related to the dimensionality of the observations?
We cannot answer this from our experiments. All our beliefs are over one- or two-dimensional states, with 100 particles (50 in Odd-Even), so input dimensionality does not vary enough to test the relationship. The Conclusion lists higher-dimensional problems as future work.
Q3. Could online representation learning improve performance, instead of offline pretraining alone?
We have not tested it, and we now say so in the paper (Section 2 and the Conclusion). It is a natural next step. A reconstruction or prediction loss trained jointly with PPO would remove the need for a separate data-collection stage. Our results suggest one caution for that direction. The choice of auxiliary target matters more than when it is trained, since a reconstruction target helped on some tasks and not on others.
Q4. Evaluation on a broader range of control tasks, such as those of Abdulsamad et al., and comparison against more existing methods.
See W1b. On comparisons with existing methods, our aim is to compare encodings under one fixed pipeline, so we deliberately do not compare end-to-end systems that also change the filter or the learning algorithm. Doing so would reintroduce the confound the study is designed to remove. The representations of the closest prior systems are included as encoders in our grid. These are the particle mean and covariance (the summary used by BetaZero-style methods), MGF-type scalar features (DPFRL, in CGF form), and the Set Transformer (Abdulsamad et al.).
Part 1 of 2
Thank you for the review and for the pointers to related work. Section and appendix numbers refer to the revised paper, where changes are shown in orange.
W1a. The encoders have been studied before, and the Set Transformer was already applied to particle beliefs by Abdulsamad et al.
We agree that the architectures are not new, and the paper does not claim them. We now cite Abdulsamad et al. in the Introduction and in Section 2. Their work proposes a policy-optimization method and encodes the belief of a fixed particle filter with a Set Transformer as one component of it. The encoder is not varied or studied. The same holds for DPFRL (a learned filter with MGF features) and Belief Embedding Tree Search (a planner with a learned embedding). Section 2 now states the difference in one place: in these methods the encoding is introduced together with a new filter, planner, or learning algorithm, so its own contribution is not isolated. We hold the particle filter and the RL procedure fixed and vary only the encoder and how it is trained.
What this comparison adds:
W1b. The evaluation is limited, and broader comparisons and more control tasks are needed.
We would like to put the size of the evaluation in context. The study covers eight tasks from four environment families: ten representations under end-to-end training, and five trainable encoders under four pretraining conditions with frozen and fine-tuned transfer, with five PPO seeds per condition. For reference, closely related work evaluates on suites of similar or smaller size: Abdulsamad et al. on four POMDP control tasks, and Belief Embedding Tree Search on three domains in six configurations. As in any empirical study, our conclusions hold for the tasks and the RL algorithm we tested, and the Conclusion says so explicitly.
On the tasks of Abdulsamad et al.: their light-dark and triangulation tasks test active information gathering, and this is the category our suite covers most heavily, with four of eight tasks (Car-Flag and the three Ant Tag variants, Section 5.1). Light-dark has the same structure as our Car-Flag: the agent must first travel to an informative region and only then commit to the goal. In light-dark that region is the light, and in Car-Flag it is the region that reveals the rewarding end. Triangulation requires the agent to move so that its observations become informative about the target, as do our Ant Tag variants, where the target is observed only within a sensing radius. Their pendulum and cart-pole tasks hide only the velocities, and the authors note that for them "active information gathering is not critical". We expect such tasks to separate belief encoders least, so we did not prioritize them. Our remaining four tasks are diagnostics built to isolate one belief property each.
We could not add new environments within the discussion period. Evaluating on further domains, including higher-dimensional problems, is future work.