acceptodds
Under review as a conference paper at ICLR 2027

Not Every Observation Should Update Belief Equally

Abstract

Partially observable reinforcement learning requires agents to maintain an internal state from incomplete observations. Although recurrent agents can attenuate incoming information through learned gates, revision strength is typically learned only implicitly through end-to-end control. Yet observation surprise and decision relevance are distinct: a surprising observation may be irrelevant, while a plausible cue may alter the preferred action. We introduce BeliefGate, which makes assimilation strength an explicit decision variable supervised by the downstream control utility of alternative belief revisions. During training, paired interventions compare candidate revision strengths from the same simulator state under matched future randomness, and the intervention selected by penalized utility supervises a continuous deployment-time gate. We evaluate diagnostic POMDPs with matched-surprise and same-token context tests, together with long-memory and observation-shift benchmarks. BeliefGate achieves the strongest overall and unseen-shift aggregates, while specialized memory baselines remain stronger on some memory-centric tasks. It also provides the strongest non-oracle relevance separation and AUROC under matched surprise, with the smallest relative degradation across observation shifts. Representation probes further show stronger decision-relevant content and lower distractor retention. These results frame observation assimilation as a decision-relevant control problem rather than a generic filtering operation.

Primary area:
reinforcement learning
Keywords:
partially observable reinforcement learning, belief representation, observation assimilation, recurrent agents, decision relevance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fKkvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BeliefGate, which uses counterfactual rollouts to supervise how strongly a recurrent agent updates its hidden state after receiving an observation. The authors report improvements on partially observable RL benchmarks, particularly under observation shifts.

Critical strengths

The motivation is reasonable, and the matched-surprise diagnostics and ablations are useful attempts to isolate the effect of the proposed gate.

Critical weaknesses
  1. Unfair computational comparison. BeliefGate uses four additional simulator steps per on-policy training step and 2.36× the training time of GRU-PPO (Tables 1 and 12). However, the baselines are not evaluated under matched total interaction or compute budgets. It is therefore unclear how much of the improvement comes from the proposed supervision rather than additional training resources.

  2. Limited validation of decision relevance. The main mechanism evaluation (Section 4.2) relies on oracle relevance labels from controlled environments. More importantly, relevance is defined by whether the optimal action changes, which does not necessarily capture an observation's long-term decision value. An observation can be informative without changing the current optimal action. It is therefore unclear whether the reported relevance metrics genuinely support the claimed utility-aware belief updates, or whether the findings generalize beyond the constructed diagnostic tasks.

  3. Poor figure quality. The figures are poorly presented. Text is often too small or too large, font sizes are inconsistent, and some labels and elements overlap, especially in Figure 1. The figures are also rasterized rather than provided as vector graphics, reducing their readability. This is below the expected presentation standard for an ICLR submission.

Questions for authors
  1. How does BeliefGate compare against baselines under matched total simulator interactions or wall-clock budgets?

  2. Can the authors provide confidence intervals and validate the utility-supervision targets on the non-synthetic benchmarks?

Reviewer_8U1WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces BeliefGate, a method for RL in POMDPs that lets an agent learn how much to revise its internal belief when receiving new observations. It combines a recurrent belief prior with an observation-dependent proposal, using a learnable scalar gate αt∈[0,1]\alpha_t \in [0,1] to control the strength of each update. During training, the agent compares different revision strengths based on their effect on future returns and uses the best-performing one as a supervision target. At deployment, the gate updates the belief in a single pass.

Empirically, BeliefGate achieves the best overall results and performs well under unseen observation shifts. It also better distinguishes relevant from irrelevant observations wirh matched surprise and shows the slowest performance degradation under observation shifts. However, specialized memory baselines remain stronger on some memory-centric tasks.

Critical strengths

Clear conceptual contribution and well-motivated problem. The distinction between observation surprise and decision relevance is an interesting insight for learning in POMDPs. Indeed, a surprising observation can be irrelevant while a plausible cue can change which action is best. This distinction is largely overlooked in prior work. Making belief revision an explicit decision variable and training it based on control utility is a meaningful contribution, even though the method builds on familiar gated-recurrence mechanisms.

The main idea is clear and consistently developed. The paper argues that agents should revise their beliefs based on the expected impact on future decisions, rather than simply on how novel an observation is. This idea carries through the method and the experiments. The use of downstream control utility to supervise revision strength is particularly interesting and differs from approaches that rely on privileged information or teacher-student supervision.

Efficient at deployment. Although the method adds computational overhead during training, its test-time cost seems negligible.

Critical weaknesses

The paper is somewhat dense. It targets an expert RL audience, and some details have to be pieced together across sections. A short worked example and explicit pseudocode for the training loop would make the method easier to follow.

Uncertainty quantification is missing. Despite using ten final seeds and paired evaluation, the paper reports no standard deviations, confidence intervals, IQRs, or significance tests for any of the metrics. Reporting these would help readers to judge whether the observed differences are robust or fall within the range of experimental noise.

Training overhead. The 2.36x training overhead from evaluating paired branches is a real cost. The authors report it transparently, and deployment latency remains low. Still, the paper would benefit from clarifying whether this overhead is inherent to the method or partly an implementation choice, and from discussing how the approach might scale to larger tasks.

Scope and generalizability. Two of the three environment sets are constructed by the authors, as are the reference policies used in them. This means that some of the evidence for BeliefGate's superior performance comes from a self-designed evaluation setup, while the memory-centric tasks are drawn from the existing POPGym suite [1]. Evaluation on a broader range of established benchmark environments without author-trained reference policies would provide stronger evidence that the benefits generalize.

The branch utility estimator is not analyzed in much detail. The paper does not characterize the variance and bias of U^t(β)\hat U_t(\beta) with M=2M=2 paired seeds and H=8H=8 steps. It also does not discuss whether the selected target β⋆β^\star can be identified from the gate's inputs, or provide a justification for the penalty κ\kappa beyond intuition. An error analysis of this estimator and a sensitivity analysis of κ\kappa would strengthen the paper.

The supervision target changes during training. The preferred revision strength is defined relative to the current policy and target networks, which themselves evolve throughout training. The paper does not examine whether these targets stabilize or whether targets generated early in training are meaningful. To address this, the authors could report target stability over training.

Minor points that did not affect my overall recommendation:

  • Some figures appear to have relatively low resolution and may not be vector graphics.
  • Figure 1 illustrates the training and deployment procedures but is not discussed in much detail in the text. In Figure 2, some curves overlap with the in-figure labels.
  • There are line breaks within inline equations, for example in Line 165.

Overall, these concerns currently outweigh the strengths of the paper. However, I am open to revising my assessment based on the authors' clarifications and any changes to the manuscript during the rebuttal phase.


[1] https://arxiv.org/abs/2303.01859

Questions for authors
  • Given that training takes more than twice as long, is BeliefGate practical for larger environments or longer episodes? Could approximations, such as evaluating fewer candidate revision levels or sparse probing, reduce the cost without trading much performance?
  • Given that β⋆β^\star is selected by taking the argmax over estimates based on only M=2M=2 paired-seeds, how large is the resulting selection bias in practice? Could the authors test this with a larger MM to determine whether optimistic target selection contributes to the observed gains?
  • β\beta is chosen from a discrete set of candidate revision strengths. How sensitive is performance to this discretization, and what happens when the best revision strength lies between two candidate values? Could the method be extended to allow continuous values of β\beta?
  • Do the utility-supervision targets stabilize during training, or do they drift as the policy and target networks change? Can the authors provide evidence of target stability over the course of training?
Reviewer_WG6POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BeliefGate, which learns how strongly each observation should update a recurrent state in partially observable RL. During training, paired simulator rollouts compare candidate update strengths and provide supervision for the gate. At test time, the gate runs without branching.

Critical strengths

The paired interventions give the supervision a clear meaning. Each branch changes only the current update strength, starts from the same simulator state, and uses matched randomness. Meanwhile, the resulting policy requires no branching at deployment.

Critical weaknesses
  1. The paper does not prove that its supervision is appropriate when the hidden state remains uncertain. Training selects a preferred update at each simulator state, while the deployed gate must act on incomplete information. States that the gate cannot distinguish may favor different updates. Regressing these choices need not produce the update with the highest expected penalized utility given the gate's inputs. The target audit and aggregate utility regret do not directly address this case.
  2. The reported performance advantage has not been established at comparable training cost. BeliefGate uses four additional simulator steps per base interaction and takes 8.5 hours to train, compared with 4.1 hours for the PPO-only gate.
Questions for authors
  1. What evidence shows that the supervision remains appropriate when the agent cannot distinguish hidden states that require different updates?
  2. Do the gains remain against the PPO-only gate or a strong memory baseline when training time is matched?
  3. How does (12) treat cues that leave the current optimal action unchanged but are necessary for later decisions?
  4. Since the proposal and gate are learned jointly, how to separate their contributions when interpreting the gate's relevance to the task?
  5. The utility targets depend on the current policy. Can the gate recover after downweighting a useful cue that the policy has not yet learned to use?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.