acceptodds
Under review as a conference paper at ICLR 2027

Particle Belief Representations for Reinforcement Learning under Partial Observability

Abstract

Particle filters flexibly represent complex, multimodal beliefs but expose them as weighted, unordered sets incompatible with conventional fixed-dimensional policies. We study particle-belief encodings as inputs to reinforcement-learning policies while holding belief estimation and policy optimization fixed. We compare statistical summaries, cumulant-generating-function (CGF) features, and permutation-invariant set encoders. Trainable encoders are learned end-to-end or pretrained through belief reconstruction or task-specific prediction. Across eight tasks, pooling-based set encoders and gradient CGF features are reliable end-to-end, while the Set Transformer achieves the highest performance with appropriate pretraining. Task-specific pretraining helps when its target captures decision-relevant belief structure, whereas reconstruction and latent alignment provide more selective benefits. These findings provide practical guidance for selecting and training encoders for particle-filter beliefs in continuous-space partially observable Markov decision processes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
POMDPs, Particle Filters, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jVXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses how to effectively encode the outputs of particle filters under partial observability. The authors consider several encoding approaches, including statistical features, cumulant-generating functions, and neural networks, while also investigating the effects of pretraining on the encoders.

Critical strengths

Major Strengths

Particle filters play an important role in reinforcement learning under partial observability, and the research direction explored by the authors is helpful for addressing this problem.

Minor Strengths

The paper conducts experiments with five random seeds, increasing the reliability of the results.

Critical weaknesses
  1. The encoding methods employed in this paper have already been studied in previous research, raising concerns about the novelty of the paper. In particular, architectures such as Set Transformer have already been applied in prior work [1]. Furthermore, the experimental evaluation is limited, and broader comparisons with existing methods and more control tasks are needed to strengthen the paper's contributions.

[1] Abdulsamad, Hany, Sahel Mohammad Iqbal, and Simo Sarkka. "Sequential Monte Carlo for Policy Optimization in Continuous POMDPs." Advances in Neural Information Processing Systems 38 (2026): 41241–41269.

  1. The discussion of pretraining is insufficient. The authors show that neural network-based encoders, such as Set Transformer, can achieve better performance after pretraining, which suggests an issue related to sample efficiency. However, representation learning for reinforcement learning has already been extensively studied [2, 3]. The paper only considers offline pretraining, although offline data can be difficult to obtain.

[2] Schwarzer, Max, et al. "Data-Efficient Reinforcement Learning with Self-Predictive Representations." arXiv preprint arXiv:2007.05929 (2020).

[3] Zheng, Ruijie, et al. "TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning." Advances in Neural Information Processing Systems 36 (2023): 48203–48225.

Questions for authors
  1. Can the authors provide a more detailed analysis of the mechanisms behind the performance differences among various encoders, including the effects of parameter scales, network depths, and other architectural factors?

  2. Is the sample efficiency issue associated with neural network-based encoders related to the dimensionality of observations? Can the authors provide further analysis of this relationship?

  3. Can the authors investigate whether incorporating online representation learning could improve task performance, rather than relying solely on offline pretraining, as explored in related work [2, 3]?

  4. Can the authors provide additional evaluations on a broader range of control tasks, such as environments similar to those in [1], where some observable state information is masked to introduce partial observability, along with comparisons against more existing methods?

2 comments
AuthorsOpenReview ↗

Part 1 of 2

Thank you for the review and for the pointers to related work. Section and appendix numbers refer to the revised paper, where changes are shown in orange.

W1a. The encoders have been studied before, and the Set Transformer was already applied to particle beliefs by Abdulsamad et al.

We agree that the architectures are not new, and the paper does not claim them. We now cite Abdulsamad et al. in the Introduction and in Section 2. Their work proposes a policy-optimization method and encodes the belief of a fixed particle filter with a Set Transformer as one component of it. The encoder is not varied or studied. The same holds for DPFRL (a learned filter with MGF features) and Belief Embedding Tree Search (a planner with a learned embedding). Section 2 now states the difference in one place: in these methods the encoding is introduced together with a new filter, planner, or learning algorithm, so its own contribution is not isolated. We hold the particle filter and the RL procedure fixed and vary only the encoder and how it is trained.

What this comparison adds:

  • Gradient CGF features, which DPFRL does not use, match or outperform the scalar features on all eight tasks (Section 6, e.g., 75% against 58% success on Ant Tag, 36% against 17% on Lantern Ant Tag), with at most 192 trainable parameters.
  • Under end-to-end training, the pooling-based encoders and gradient CGF features are reliable, while the Set Transformer, the choice in Abdulsamad et al., reaches its ceiling only from a pretrained start on several of our tasks.
  • The pretraining objective matters more than the transfer mode, and an encoder that reconstructs the belief well is not necessarily a good input for control.

W1b. The evaluation is limited, and broader comparisons and more control tasks are needed.

We would like to put the size of the evaluation in context. The study covers eight tasks from four environment families: ten representations under end-to-end training, and five trainable encoders under four pretraining conditions with frozen and fine-tuned transfer, with five PPO seeds per condition. For reference, closely related work evaluates on suites of similar or smaller size: Abdulsamad et al. on four POMDP control tasks, and Belief Embedding Tree Search on three domains in six configurations. As in any empirical study, our conclusions hold for the tasks and the RL algorithm we tested, and the Conclusion says so explicitly.

On the tasks of Abdulsamad et al.: their light-dark and triangulation tasks test active information gathering, and this is the category our suite covers most heavily, with four of eight tasks (Car-Flag and the three Ant Tag variants, Section 5.1). Light-dark has the same structure as our Car-Flag: the agent must first travel to an informative region and only then commit to the goal. In light-dark that region is the light, and in Car-Flag it is the region that reveals the rewarding end. Triangulation requires the agent to move so that its observations become informative about the target, as do our Ant Tag variants, where the target is observed only within a sensing radius. Their pendulum and cart-pole tasks hide only the velocities, and the authors note that for them "active information gathering is not critical". We expect such tasks to separate belief encoders least, so we did not prioritize them. Our remaining four tasks are diagnostics built to isolate one belief property each.

We could not add new environments within the discussion period. Evaluating on further domains, including higher-dimensional problems, is future work.

AuthorsOpenReview ↗

Part 2 of 2

W2. The discussion of pretraining is insufficient, representation learning for RL is well studied, and only offline pretraining is considered.

We agree that this literature was missing. Section 2 now places our pretraining within it. In RL from images, representation learning can be decoupled from policy learning by pretraining the encoder and using it frozen (ATC, Stooke et al.), which is the route we follow for particle beliefs. Alternatively, self-supervised losses can be trained jointly with the RL objective (SPR, TACO), which we do not study. The Conclusion lists jointly trained objectives as future work.

We add two clarifications on the points raised.

  • On offline data being hard to obtain, reconstruction pretraining needs only beliefs, with no rewards and no labels. For Faint and Diffuse Beacon these are drawn from the task's initial-layout distribution without any rollouts. For the other tasks they come from scripted rollouts (Appendix H). Task-specific pretraining does need labels read from simulator state, and Appendix H now says so.
  • On sample efficiency, we agree with this reading for end-to-end training. Within our RL budget, the PPO objective alone does not train the Set Transformer to a useful encoding on several tasks, while a suitable pretrained start does. We did not test whether a longer RL budget would close the gap. One qualification is that a head start alone is not enough. On Faint and Diffuse Beacon, reconstruction pretraining leaves the Set Transformer at 13% success while task-specific pretraining brings it to 88% and 91%. What the pretraining objective teaches the encoder matters, and not only that the encoder is pretrained.

Q1. Mechanisms behind the differences, including parameter scale, depth, and architecture.

Appendix F now reports the size of every representation. The three set networks are matched in size (107,840 parameters for Deep Sets and PointNet, 109,640 for the Set Transformer on the 2-D tasks) and in output width (64). The differences between them are therefore not explained by parameter count. The architectural difference we point to in Section 6 is that the Set Transformer models interactions between particles before pooling, while Deep Sets and PointNet encode each particle independently. This matches where the Set Transformer leads after pretraining (tasks that require comparing particle clusters). We did not vary depth or width, and we do not claim an analysis of those factors.

Q2. Is the sample-efficiency issue related to the dimensionality of the observations?

We cannot answer this from our experiments. All our beliefs are over one- or two-dimensional states, with 100 particles (50 in Odd-Even), so input dimensionality does not vary enough to test the relationship. The Conclusion lists higher-dimensional problems as future work.

Q3. Could online representation learning improve performance, instead of offline pretraining alone?

We have not tested it, and we now say so in the paper (Section 2 and the Conclusion). It is a natural next step. A reconstruction or prediction loss trained jointly with PPO would remove the need for a separate data-collection stage. Our results suggest one caution for that direction. The choice of auxiliary target matters more than when it is trained, since a reconstruction target helped on some tasks and not on others.

Q4. Evaluation on a broader range of control tasks, such as those of Abdulsamad et al., and comparison against more existing methods.

See W1b. On comparisons with existing methods, our aim is to compare encodings under one fixed pipeline, so we deliberately do not compare end-to-end systems that also change the filter or the learning algorithm. Doing so would reintroduce the confound the study is designed to remove. The representations of the closest prior systems are included as encoders in our grid. These are the particle mean and covariance (the summary used by BetaZero-style methods), MGF-type scalar features (DPFRL, in CGF form), and the Set Transformer (Abdulsamad et al.).

Reviewer_R8z7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies various approaches for summarizing beliefs represented as particle sets into a fixed-size vectors suitable as inputs to neural networks. Maintaining rich beliefs and summarizing them into fixed-size vectors is an important step for partially observable RL. The explored approaches include simple statistics, cumulant-generating functions and learned set-encoders, such as DeepSets and Set Transformer; trainable encoders are trained end-to-end or pre-trained. The performance of each variant is measured through the downstream performance of a PPO agent that conditions its policy on the provided belief representation in several manually constructed environments that capture properties that arise in POMDPs.

Critical strengths

The paper studies an important component in a highly active area of POMDP research. It provides a strong motivation, and discusses and delineates itself from the prior work well.

Critical weaknesses

The way set encoders were pretrained seems problematic. As I understood it from the paper, the decoder is trained to map the embedding to N uniformly weighted particles regardless of what belief entered the encoder (and thus what belief the embedding represents). Plus, the encoders were trained only on a single seed.

The experiments only contain very specific, artificial environments and the beliefs are sometimes manually created which doesn’t correspond with how a standard particle filter would represent the beliefs during inference.

Although the reproducibility statement promises additional details in Appendices C, D and E, there are only environment hyperparameters. Other architecture, encoder pre-training and PPO training hyperparameters and details are missing and the authors did not provide the code in the supplementary materials. Overall, the paper does not contain enough details to understand the experimental setup and allow reproduction of the results. Additionally, the whole Appendix provides lots of tables with very little discussion and Appendix G feels unfinished. Appendix A and Figure 1 are never referenced in the text.

Lastly, approaches that combine learned set encoders and generative models have recently been proposed as (Generative) Distribution Embeddings [1, 2] and used in an online POMDP solver [3], but the paper does not mention either of those.

[1]: Solinas, Christopher, et al. "Neural Bayesian Filtering." arXiv preprint arXiv:2510.03614v1 (2025). [2]: Fishman, Nic, et al. "Generative Distribution Embeddings: Lifting autoencoders to the space of distributions for multiscale representation learning." Advances in Neural Information Processing Systems 38 (2026): 24644-24685. [3]: Solinas, Christopher, et al. "Belief Embedding Tree Search." Reinforcement Learning and Video Games Workshop @ RLC 2025, 2025.

Questions for authors
  • How did you collect a dataset of beliefs used for pre-training decoders?

  • If collected with a different policy than what PPO produces (which I assume is the case), how did you deal with off-policy data?

  • Have you tried training the encoder to reconstruct beliefs jointly with PPO?

  • The authors motivate learned belief representation as suitable for downstream tasks, such as diagnostics.

  • How does end-to-end training affect the representational quality of encoded beliefs?

2 comments

Thank you for the detailed review. The revision adds the experimental details the review asks for, in three new appendices. Section and appendix numbers refer to the revised paper, where changes are shown in orange.

W1a. The decoder outputs N uniformly weighted particles regardless of the input belief.

The decoder's output weights are uniform, but the loss compares that output with the weighted input belief, so the input weights are used. Appendix H now explains this. The loss is the debiased Sinkhorn divergence between the decoder's N equally weighted points and the input particles carrying their filter weights. The input weights enter as the target marginal of the transport problem, and the input belief is not resampled. A weighted belief is therefore reconstructed by an equally weighted set that places more points where the input has more mass, which is what resampling does. The decoder is conditioned only on the encoder's output, so different beliefs produce different point sets.

W1b. The encoders were pretrained on a single seed.

This is correct and deliberate, and Section 5.2 states it. For each encoder and objective, one pretrained checkpoint initializes all five PPO seeds. Our question is how a given encoder performs as a policy input, so we hold the encoder fixed and report the variation of policy optimization. Drawing a new encoder for each PPO seed would mix two sources of variance, and a difference between conditions could no longer be attributed to either. The consequence is that our error bars do not include variation between pretraining runs, and we do not claim that they do.

We understand the reviewer's concern to be that a single checkpoint could be unrepresentative. An unrepresentative checkpoint would not explain our conclusions, because the same result appears in pretraining runs that are independent of each other. The Set Transformer was pretrained separately for Faint Beacon and for Diffuse Beacon. In both, reconstruction pretraining gives 13% success and task-specific pretraining gives 88% and 91%. On Diffuse Beacon, the separately pretrained Deep Sets and PointNet encoders move in the same direction (14% to 63%, and 19% to 79%). These gaps are tens of points, while the standard error across PPO seeds in these cells is about one point.

W2. The environments are artificial and some beliefs are manually created.

The eight tasks are of two kinds, and the paper uses them for different purposes.

  • Car-Flag and the three Ant Tag variants are adapted from the robot POMDP domains of Nguyen (2021), cited in Section 5.1, and run a particle filter that updates from observations.
  • The three beacon tasks and Odd-Even are diagnostic tasks. Section 5.1 states that the beacon belief is a particle representation of the long-run signal distribution, a construction that "isolates belief representation from estimation error", and that the Odd-Even belief is the exact posterior with one particle per candidate. We built them this way on purpose. Each isolates one property of a belief (coverage, cluster mass, cluster spread, parity-constrained support) so that an encoder's failure can be attributed to the representation and not to filtering error.

W3. Experimental details, code, and the appendix.

The revision addresses each of these points.

  • Appendix F gives the architecture of each encoder, and a table of output widths and trainable-parameter counts. The three set networks have about 108,000 parameters each, and the learned CGF probes have at most 192.
  • Appendix G gives a table of all PPO hyperparameters per task group. They are the same for every encoder and training regime within a task.
  • Appendix H gives the pretraining data, the reconstruction loss, the prediction heads and the source of their labels, the alignment schedule, and a table of pretraining settings.
  • The ablation appendix (now Appendix I) is complete, with final plots and captions, and is referenced from Section 5.2.
  • Figure 1 is referenced from the Introduction. The former Appendix A was removed. Every appendix is now referenced from the main text.
  • The reproducibility statement now points to the correct appendices.

On code, we will release the code and the trained model weights upon publication, as the reproducibility statement says. In the meantime, Appendices F, G, and H give the architectures, hyperparameters, and pretraining settings needed to reimplement the study.

W4. Generative Distribution Embeddings, Neural Bayesian Filtering, and Belief Embedding Tree Search are not mentioned.

All three are now cited and discussed in Section 2. We knew of Generative Distribution Embeddings and Neural Bayesian Filtering when writing the paper. We left them out because they study estimation, meaning how well a distribution or a hidden state is recovered, and not control, which is our subject. We now think the paper should state that difference explicitly, and Section 2 does so.

We were not aware of the workshop paper on Belief Embedding Tree Search when we submitted. The arXiv paper of the same title appeared on 10 October 2026, after the ICLR submission deadline, and that is the version we cite. It replaces the particle filter and the belief encoding together. Its analysis attributes the gains to the regenerated particles on RockSample and to the learned embedding on PocMan. However, in every comparison with a particle-filter baseline the filter and the embedding change together, so how much of the benefit comes from better filtering and how much from a better embedding of the belief is not measured separately. Our work investigates exactly that. For each task we keep the particle filter fixed and change only how its particles are embedded for the policy, so any difference in control performance is due to the embedding. Neural Bayesian Filtering was available only on arXiv when we submitted. It has since been accepted at NeurIPS 2026, with decisions announced one day before the ICLR deadline.

The ICLR reviewer guidelines encourage, but do not require, authors to cite work that is not published in peer-reviewed proceedings, and they state that the lack of comparisons to such work "cannot be a basis for rejection". We agree that these references belong in the paper, and the revised paper discusses all three.

Q1. How was the pretraining dataset collected?

Appendix H gives the source and number of beliefs for every task. A brief list follows.

  • The Ant Tag variants use 100,000 beliefs (38,271 for Hideout Ant Tag) from scripted rollouts that mix four behaviors, namely walking between sampled waypoints (probability 0.5), pursuing the belief mean (0.3), pursuing the target with the target always visible (0.1), and random actions (0.1).
  • Beacon Sweep uses 185,009 beliefs from scripted rollouts (60% pursue an active beacon, 40% random walk).
  • For Faint and Diffuse Beacon, reconstruction pretraining uses beliefs drawn from the task's initial-layout distribution, without rollouts.
  • Odd-Even uses 136,400 beliefs from random actions, since the belief there does not depend on the actions.

Q2. How did you deal with off-policy data?

Pretraining is not reinforcement learning. The encoder is trained by reconstruction or supervised prediction on a fixed set of beliefs, and no rewards, returns, or value estimates are involved. The off-policy problem of RL therefore does not arise. The relevant question is coverage, that is, whether the pretraining beliefs resemble those the PPO policy later visits. Two points address it.

  • The scripted behaviors (Appendix H) include pursuing the belief mean and pursuing the target, which produce beliefs of the kind a competent policy reaches.
  • If the mismatch mattered, an encoder that keeps adapting to the policy's own beliefs should do systematically better than one that cannot. It does not. Across the 130 pretrained conditions, fine-tuned encoders are better than frozen ones by only 0.02 in success on average, and the larger differences in single conditions go in both directions.

Q3. Have you tried training the encoder to reconstruct beliefs jointly with PPO?

No. All our pretraining happens before RL. The revision says so in Section 2, where we now cite work that trains self-supervised losses jointly with the RL objective (SPR, TACO), and lists jointly trained objectives as future work in the Conclusion.

Q4. How does end-to-end training affect the representational quality of the encoded beliefs?

We did not measure this, and we agree it is a natural follow-up. What our results show is the reverse direction. Representational quality as measured by reconstruction does not guarantee control performance (the reconstruction-pretrained Set Transformer reaches 13% success on Faint and Diffuse Beacon, the task-pretrained one 88% and 91%). Measuring how well an encoder trained only by PPO reconstructs the belief would complete the picture, and we leave it to future work.

Reviewer_ZpLmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies different encodings of weighted particle sets that approximate beliefs in POMDPs. These encodings are fixed size and suitable for input into RL policies. The tested encoders include simple Gaussian statistics, cumulant-generating-function (CGF) features, and three set-invariant networks are evaluated across 8 tasks and trained either end-to-end or pre-trained. The stated contributions are the empirical study of existing methods and task suite, rather than any of the methods or new theory regarding particle belief representations.

Critical strengths

The study varies only the belief encoder and its training, with the particle filter, RL algorithm, and policy architecture held fixed. This isolates the effect of the encoder on control performance and makes the central question asked in the paper well-posed.

Critical weaknesses

Missing related work. The paper does not engage with the literature on distribution embeddings [1], which studies exactly the map from a sample set to a fixed-size vector, including kernel mean embeddings and filtering over distribution embeddings as done in [2]. These omissions are not just missing citations because many of the encoding methods tested in this paper have also been tested in prior work. This is a non-exhaustive list of the missing related work.

[1] Fishman, Nic, et al. "Generative Distribution Embeddings: Lifting autoencoders to the space of distributions for multiscale representation learning." Advances in Neural Information Processing Systems 38 (2026): 24644-24685.

[2] Solinas, C., Haluska, R., Sychrovsky, D., Timbers, F., Bard, N., Buro, M., ... & Bowling, M. (2025). Neural Bayesian Filtering. arXiv preprint arXiv:2510.03614.

Limited contribution The paper states its contribution as an empirical study of existing methods, but its conclusion to the central question it poses of "What are the most effective particle-belief encodings for reinforcement learning, and how should they be constructed??" is basically that it depends on the task. I don't feel I learned a lot about how to encode particle approximations of beliefs from this paper, so I don't believe that it's contribution, as stated, is strong enough. I'm also not necessarily convinced that under this empirical setup, any results would generalize beyond this set of tasks and encoders.

Questions for authors
  • Is the Set transformer failure in the end-to-end case a result of unstable optimization that pretraining compensates for? Would it still fail with more tuning? Some of the prior work mentioned in the weaknesses uses Set transformers effectively in end-to-end settings.
2 comments
AuthorsOpenReview ↗

Part 1 of 2

Thank you for the review. We respond to each point below. Section and appendix numbers refer to the revised paper, where changes are shown in orange.

W1. Missing related work on distribution embeddings.

We agree that this literature should have been discussed. We knew of Generative Distribution Embeddings and Neural Bayesian Filtering when writing the paper and left them out because they study estimation, meaning how well a distribution or a hidden state is recovered, and not control, which is our subject. That was the wrong call. The paper should have stated the difference explicitly, and the revision now does so in Section 2. The additions are as follows.

  • Generative Distribution Embeddings (Fishman et al.) and Neural Bayesian Filtering (Solinas et al.) are discussed in the paragraph on set encoders. Neural Bayesian Filtering is also cited in the Introduction.
  • Kernel mean embeddings (Muandet et al.) are cited in the paragraph on moment- and cumulant-generating functions, because a CGF evaluated at finitely many probe points is a special case.
  • We also added Belief Embedding Tree Search (Solinas et al.), which uses such embeddings inside a tree-search planner. It was released on arXiv in October 2026, after the ICLR submission deadline.

On the concern that these encoders "have also been tested in prior work", we agree that the architectures are not new, and the paper does not claim them. What differs is the question asked and what is held fixed.

  • Fishman et al. and Solinas et al. (Neural Bayesian Filtering) evaluate how accurately the distribution or the hidden state is recovered. We evaluate the return of a policy trained on the encoding. Our results show that these two criteria can disagree. On Faint Beacon and Diffuse Beacon, an encoder pretrained to reconstruct the belief gives the Set Transformer 13% success, while the same encoder pretrained on a task-specific target reaches 88% and 91%.
  • In DPFRL, Belief Embedding Tree Search, and Abdulsamad et al., the encoding is introduced together with a new filter, planner, or learning algorithm. The effect of the encoding alone is therefore not isolated. We hold the particle filter and the RL procedure fixed and vary only the encoder and how it is trained.

W2. Limited contribution, and the answer is "it depends on the task".

We accept that the original Conclusion read this way, and we have rewritten it to state three findings that hold across the task groups.

  1. Under end-to-end training, pooling-based set encoders (Deep Sets, PointNet) and gradient CGF features are the reliable choices. The Set Transformer has the highest ceiling but reaches it only from a pretrained start.
  2. The pretraining objective matters more than whether the encoder is then frozen or fine-tuned. Across the 130 pretrained conditions, fine-tuning changes success by +0.02 on average. Task-specific prediction is decisive when its target is the decision variable, and can hurt when it replaces a belief that must stay uncertain with a point estimate.
  3. Latent metric alignment gives no consistent benefit.

Two further points in the revision bear on what a reader learns.

  • Gradient CGF features, which DPFRL does not use, match or outperform the scalar features on all eight tasks (Section 6, e.g., 75% against 58% success on Ant Tag, 36% against 17% on Lantern Ant Tag). They have at most 192 trainable parameters, against about 108,000 for the set networks (Appendix F).
  • An encoding that reconstructs the belief well is not necessarily a good input for control. This is the practical message for anyone pretraining a belief encoder with a generative or reconstruction objective.

On generalization beyond these tasks, the study was designed with this concern in mind. Each diagnostic task isolates one property of a belief (multimodality, cluster mass, cluster spread, parity-constrained support), so each finding is tied to a belief property and not to one environment. This is what lets a reader judge whether a finding applies to their own task. As in any empirical study, the conclusions are established only for the tasks and the RL algorithm we tested, and the Conclusion says so.

AuthorsOpenReview ↗

Part 2 of 2

Q1. Is the Set Transformer's end-to-end failure an optimization problem that pretraining compensates for? Would it still fail with more tuning?

Our reading is that it is an optimization difficulty and not a capacity limit, and Section 6 now says so with the evidence.

  • On Beacon Sweep, all encoders use identical PPO settings (Appendix G). Deep Sets and PointNet reach 97% and 98%, and the Set Transformer reaches 21%.
  • The same Set Transformer architecture reaches 87% (frozen) and 95% (fine-tuned) after reconstruction pretraining, and 100% after task-specific pretraining. Its capacity is therefore sufficient for the task.

On Beacon Sweep we also tried a lower learning rate, fewer epochs per update, and changes to normalization. None trained the Set Transformer end to end. The failure is not a defect of our implementation, because on Ant Tag the same architecture achieves a high success rate when trained end to end (87%). We do not claim that no PPO setting could train it on Beacon Sweep. Our claim is narrower. On this task it is markedly harder to train from the reward alone than the pooling encoders, which succeed under the same PPO settings.

On the prior work the reviewer refers to, Generative Distribution Embeddings and Neural Bayesian Filtering train the encoder for estimation, not for control. The encoder is trained together with a generator on a generative loss and is judged by how well the distribution or the hidden state is recovered. No policy is learned. Our setting is control. Under end-to-end training the encoder and the policy are trained together, and the only signal is the PPO objective. Our results indicate that for the Set Transformer this signal alone was not enough on several of our tasks, most clearly Beacon Sweep, while the same encoder trains reliably on a reconstruction loss, which is the setting of those two papers and of our reconstruction pretraining. The two observations are therefore consistent.

Abdulsamad et al., whom we now cite, do train a Set Transformer encoder from a control objective. Their setting differs from ours in the policy-optimization algorithm, the tasks and their difficulty, and the belief size (32 particles against 100). The architecture can therefore be trained end to end for control in some settings. This supports reading our Beacon Sweep result as an optimization difficulty and not as a limitation of the architecture.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.