- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BeliefGate, which uses counterfactual rollouts to supervise how strongly a recurrent agent updates its hidden state after receiving an observation. The authors report improvements on partially observable RL benchmarks, particularly under observation shifts.
Critical strengths
The motivation is reasonable, and the matched-surprise diagnostics and ablations are useful attempts to isolate the effect of the proposed gate.
Critical weaknesses
-
Unfair computational comparison. BeliefGate uses four additional simulator steps per on-policy training step and 2.36× the training time of GRU-PPO (Tables 1 and 12). However, the baselines are not evaluated under matched total interaction or compute budgets. It is therefore unclear how much of the improvement comes from the proposed supervision rather than additional training resources.
-
Limited validation of decision relevance. The main mechanism evaluation (Section 4.2) relies on oracle relevance labels from controlled environments. More importantly, relevance is defined by whether the optimal action changes, which does not necessarily capture an observation's long-term decision value. An observation can be informative without changing the current optimal action. It is therefore unclear whether the reported relevance metrics genuinely support the claimed utility-aware belief updates, or whether the findings generalize beyond the constructed diagnostic tasks.
-
Poor figure quality. The figures are poorly presented. Text is often too small or too large, font sizes are inconsistent, and some labels and elements overlap, especially in Figure 1. The figures are also rasterized rather than provided as vector graphics, reducing their readability. This is below the expected presentation standard for an ICLR submission.
Questions for authors
-
How does BeliefGate compare against baselines under matched total simulator interactions or wall-clock budgets?
-
Can the authors provide confidence intervals and validate the utility-supervision targets on the non-synthetic benchmarks?