acceptodds
Under review as a conference paper at ICLR 2027

Decision-Relevant Representations under Rare Observations

Abstract

World models and predictive representations decide what to remember by how well the past predicts the future. This objective has a blind spot: a cue that is rare, unpredictable from the past, and yet decides the action—a traffic light that turns red—contributes almost nothing to an average predictive loss and is the first distinction compression discards. We show that the right currency for such cues is evidence, not frequency. We introduce coverage-calibrated Bellman quotients (CBQ), a latent-state learner that merges two histories only when their counts certify that no action consequence separates them, and returns the most compact code that passes this certificate. CBQ comes with a complete finite-sample account. Its supported -error splits into approximation bias and statistical error ; it keeps every action-flipping cue whenever , recovers the exact Bellman quotient—discarding all decision-null nuisance—under a separation condition, and turns both into fresh-rollout regret bounds. A matching obstruction shows that no learner can identify a rare decisive cue when , so in the one-stage case the evidence requirement of CBQ is tight up to logarithms. Across 288 random POMDPs, predictive objectives keep the rare cue in at most of instances at any sample size, while CBQ keeps it in all of them with a code exactly as compact as the Bellman quotient; all 4,320 runs switch on within one decade of the theory's coverage statistic , and the certificate is never violated. From pixels, next-frame and self-predictive world models lose a rare traffic light in 107 of 108 runs, whereas CBQ on a learned patch tokenizer keeps exactly the light's patch and discards the watermark entirely.

Primary area:
learning theory
Keywords:
representation learning, partial observability, state abstraction, decision relevance, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ptQEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a method called coverage-calibrated Bellman quotients (CBQ) with the aim of better dealing with decision-relevant rare cues, based on a compression of evidence, rather than frequency.

Critical strengths

None

Critical weaknesses
  • It’s not clear what advantage this provides over standard predictive modeling of control-theoretic variables. Figure 4 not clear what task this would solve in the real-world, besides be able to predict the red cue, which by definition CBQ would predict. But if the red cue is decision-relevant, then it would be picked up by a neural networks whose weights are optimized in the standard predictive modeling sense. So it's not clear what the practical advantage of CBQ is, and it seems incorrectly portrayed as a limitation of predictive modeling, when in fact it can be covered by it.

It would therefore be important to exhibit a comparison between CBQ and DQN (both with neural network encoders as in Fig. 4) but when the red cue is decision-relevant, and to show that CBQ outperforms (or is more sample-efficient) than DQN.

  • The experiments are toy so it would be good to additionally try to demonstrate the advantage of this approach (and exhibit this decision-relevant comparison between CBQ and DQN) to Craftax (or any other visually rich setting).
Questions for authors
  • The writing is not very clear and uses unconventional control-theoretic terms, for example, what is an "atom"? I don't see it defined anywhere.

  • Please address the weaknesses above

Reviewer_aMSdOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
1(−1)
Soundness
1(−1)
Read review
Paper summary

This paper studies how representations can retain rare cues that change the optimal action. It proposes coverage-calibrated Bellman quotients (CBQ), which select a representation with the fewest supported codes among candidates passing count-dependent reward and transition tests. The authors provide bounds on Q-value error and policy regret under coverage assumptions. They also give conditions for retaining rare cues and recovering the supported Bellman quotient, and establish an information-theoretic lower bound for the one-stage setting. Experiments on synthetic tabular and pixel tasks illustrate these results.

Critical strengths

The analysis separates approximation bias from sampling error and gives explicit conditions for retaining rare cues. The backward recovery argument also establishes conditions for recovering the Bellman quotient in the finite-master setting. I find this distinction useful because it clarifies which errors can be reduced with more data and which require a better abstraction. The controlled experiments examine nuisance compression, coverage, and delayed consequences. The comparison with the myopic variant is particularly useful: in the delayed-reward task, CBQ retains the cue while the myopic variant fails to do so. This shows why agreement in immediate rewards alone is not enough to justify merging states.

Critical weaknesses

My main concern is that CBQ starts from an already sufficient master representation. Assumption 6 requires identical reward means and transition laws within each atom, while AtA_t only covers subsequent merging, not information lost by the tokenizer. In Appendix B.6, the tokenizer already recovers all 24 scene states exactly. This does not justify applying the certificate unchanged to general learned tokenizers. Coverage is still required at the master state-action level, and unrestricted partition selection involves minimum clique cover.

The interpretation of the certificate is also too strong. Wider confidence radii make Eqs. (4) and (5) easier to satisfy; a supported action row with zero observations imposes no constraint on merging through those tests. Passing the tests therefore does not itself establish that a merge preserves optimal decisions. Similarly, At≥Δt/2A_t \ge \Delta_t/2 makes the retention guarantee inconclusive, but does not imply that the cue must be lost at every sample size.

The advance over Jiang et al. (2015), who already study statistical abstraction selection with finite-sample guarantees, needs to be clearer. The backward recovery result is useful, but the matching lower bound only covers the one-stage setting. Also, the Bellman quotient preserves more than the optimal action, so its recovery does not establish minimality among all decision-sufficient representations.

Finally, from my opinion, the experiments support compactness more clearly than better decision learning. In Table 7, Q regression retains the cue in 33 of the 36 runs at shared sample budgets, compared with 10 for CBQ, despite also using an average loss. Removing the watermark gives a smaller representation, but does not by itself improve control. The comparisons do not isolate confidence calibration from the tokenizer and representation constraints, so the broader claims about predictive learning need to be narrowed.

Questions for authors

Please address the concerns in Weaknesses.

Reviewer_HDy2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper deals with the problem of learning compact representation that preserves the decision relevant states in which previous methods might be failing according to the authors due to the methods incorrect loss that causes it to save predictable but decision irrelevant information, this is especially critic where a single cue has big impact on the correct action to take. To address this the paper introduces Coverage-Calibrated Bellman Quotients (CBQ), a method that compress a given initial discrete state representation by merging states whose reward and transition consequences are statistically compatible. The algorithm selects the smallest state partition based on confidence radii calibrated to the observed counts of individual state-action pairs. In the theoretical sections the authors establish finite-sample guarantees on Q-value estimation error and policy regret, conditions for preserving cues that are relevant for the correct action choosing and recovering the exact supported Bellman quotient and an information-theoretic lower bound that matches the sample complexity dependence in a restricted one stage setting. The experimental part shows the method ability to correctly keep the relevant states in POMDP environments and saving the correct pixels in a given observation.

Critical strengths
  1. Statistical approach: the paper and the described method of CBQ provides a clear systematic mechanism for deciding which states to merge rather than fixed similarity. The paper provide the theoretical ground and math to account for the uncertainty that arises especially under rare observations.
  2. Comprehensive theoretical analysis: The authors provide finite-sample bounds on Q-value approximation and policy regret, separating approximation bias from statistical estimation error. They additionally derive conditions for preserving decisive cues and recovering the exact Bellman quotient, along with an information-theoretic lower bound. Connecting state-action coverage, action gaps, and representation recovery within a unified analysis is a valuable contribution. 3)Extensive controlled experiments and ablations: The experiments investigate the effects of event rarity, sample size, coverage, merging thresholds, delayed rewards, and nuisance information. The random-POMDP benchmark and pixel-based experiments provide complementary evaluations, while the ablations help illustrate the conditions under which the proposed method succeeds or fails.
Critical weaknesses

1)Strong dependence on the initial representation: CBQ depends on the availability of finite master representation that preserves the information required for optimal decision-making with the relevant reward and dynamics. This is particularly relevant in the paper setup and described problem of identifying the relevant cues especially under rare observations. This also is marked by the pixel experiment in which CBQ operates on discrete states generated by a separately trained tokenizer, whereas the competing methods learn continuous neural representations directly from raw images. The CBQ tokenizer successfully separates all underlying traffic-light conditions before the proposed algorithm is applied. 2)Insufficient comparison against control-oriented representation learning: A central argument of the paper is that predictive objectives fail to preserve rare decisive cues. However, the authors' own experiments show that sufficiently increasing the reward-prediction weight can preserve these cues. Furthermore, the end-to-end Q-regression baseline successfully retains the traffic-light information in the pixel experiments, although it also preserves nuisance information. This might suggest that the problem isn't inherent and can be mitigated and perhaps fixed using correct choice of training objective. A fair evaluation might include comparisons against appropriately tuned or adaptive reward-weighted objectives, as well as reward-only or Q-based representation learning with explicit compression or regularization 3)Limited demonstrated downstream benefit of state compression: Although CBQ successfully produces compact representations while preserving decisive information, the experiments do not clearly establish how much this compression improves downstream decision-making compared with simpler alternatives. The uncompressed identity representation can preserve the relevant information, while end-to-end Q regression also succeeds in retaining the decisive cue. The paper would benefit from a clearer assessment of whether CBQ improves sample efficiency, planning cost, memory requirements, or policy performance relative to methods that preserve the relevant information without obtaining the exact minimal quotient. 4)Unclear novelty relative to existing state abstraction methods: The Bellman quotient and the general principle of merging states with equivalent reward and transition consequences are already known in the literature on bisimulation and MDP abstraction. Likewise, confidence-based statistical estimation is a familiar tool in reinforcement learning. Although the paper provides a substantial theoretical analysis of the proposed combination, its algorithmic novelty relative to existing statistical state-abstraction and abstraction-selection methods is unclear. A more explicit methodological comparison with prior approaches would clarify which aspects of the merging procedure and its guarantees are genuinely new.

Questions for authors

1)As described above, CBQ assumes a master representation that already preserves all decision-relevant distinctions and in the pixel experiments, the tokenizer successfully separates all underlying traffic-light conditions before CBQ is applied. How sensitive is CBQ to imperfections in this initial representation? In particular, how would the method handle a merged rare but decisive observation or duplicated states, and how realistic is the assumption of an already sufficient master representation in more complex environments? 2)The experiments suggest that increasing the reward-prediction weight can resolve the rare cue retention failure, while the end-to-end Q-regression baseline already preserves the decisive traffic-light information. Could the authors clarify what fundamental advantage CBQ offers over sufficiently tuned or adaptive reward-weighted objectives, or Q-based representation learning with explicit compression regularization? Would such stronger baselines achieve similar retention and nuisance removal? 3)Can the authors quantify improvements in sample efficiency, policy performance, planning cost, or memory requirements relative to an uncompressed representation or other methods that already preserve the relevant cues? Does the reduced state space translate into a meaningful practical benefit?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.