- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reward-free compression for state abstaction in RL. Specifically, they prove that passive structural entropy cannot distinguish different MDPs but with identical dynamics under action averaging. To address the limitation, they introduces control gain to measure the action identifiability. They also proposed to use action condition-code length as criteria for state aggregation.
The experiment show that the proposed control gain indeed distinguish what preivous method cannot. However, the rest of experiment results inconclusive.
Critical strengths
-
I appreciate that the authors are open about the reuslts that did not work well. The learned-agent study is described as inconclusive. Post-hoc analyses are clearly labelled. I find the level of transparency unusual and is of high standard.
-
The paper identifies a real limitation of passive structural entropy and illustrated with clear arugment and examples.
Critical weaknesses
-
The novelty relative to existing action-information based method need to be better justified. The idea of using action uncertainty or information gain is not novel.
-
Downstream usefulness is not established. According to the exmperiments, the action-conditioned codelength objective does not reliably produce planning-preserving partitions.
-
Practical scalability is not established for the implemented estimator and partition search. The envs used in experiments are simple with specifically designed MDP. It is not clear how this idea can scale into more realistic settings.
Questions for authors
What is the main new insight beyond existing action-conditioned mutual information and empowerment measures, particularly Seitzer et al.’s uniform-policy causal influence measure?
Given the unsuccessful abstraction-planning test and inconclusive learned-agent study, what useful role for the diagnostic is established by the current results?
Response to Reviewer oRge: what is new beyond known action-information measures, and what use the paper shows
Thank you for the review. We agree with all three weaknesses.
Q1 and W1: what is new beyond action-conditioned mutual information and empowerment, in particular Seitzer et al. The information quantity is not new. For one state, our control gain is the mutual information between the action and the next state under uniformly random actions. Seitzer et al. (NeurIPS 2021) define this conditional mutual information for a state and one component of the next state, also under uniform actions, estimate it with a learned model, and use it to improve exploration and off-policy learning on robotic manipulation tasks. We did not cite them and should have. The link to empowerment, which Seitzer et al. also note for their measure, is Proposition 2: the quantity is a lower bound on empowerment averaged over the stationary state distribution, with equality when uniform actions achieve the channel capacity.
So what is new in the paper is not the measure. What is specific to the paper is a statement about structural entropy: its codelength for an MDP, under any encoding tree, depends only on the action-averaged transition kernel (Proposition 1). It cannot tell apart two MDPs that have the same averaged kernel, however differently their actions select successors. Beyond that the paper contains two things:
Q2 and W2: what useful role is established. Only measurement, and only on small constructed MDPs. Where the true value is known, the calibrated estimate, with every state as its own block, matches it: at each of the five settings per family (f in the paper) the mean estimate is within about 0.015 bits of the true value. A use for choosing abstractions is not established, as you say. In the planning test, partitions chosen with the action-aware code beat partitions chosen with the action-blind code in at most 2 of 5 seeds at each partition size (1 of 5 when ties are not counted). Across all 45 partitions (these two codes and a third objective added afterwards; three sizes; five seeds; a post-hoc analysis), the action information a partition keeps is not significantly related to the return it keeps (rank correlation 0.17, p = 0.26; Section 5.2). Across 15 trained agents, the rank correlation between the calibrated estimate and planning success is 0.18 (p = 0.54; Section 5.3).
W3: scale. Agreed. Every positive result is on a tabular MDP of at most 40 states, built for the purpose. The one continuous case reduces the latent space to 40 clusters and the actions to four bins; there the calibrated values are −0.004 to 0.006 bits, above twice the null's standard deviation for 2 of the 15 agents. The paper does not show that the estimator or the partition search works at a realistic scale (Section 7).
Three points of our own. First, the caption of Figure 1 says "Averaged over any fixed behaviour policy the two are the same Markov chain", and the label inside the figure implies the same. That holds for uniformly random actions, which the experiments use (Section 2), not for an arbitrary policy. Proposition 1 is stated for a given averaged kernel and is not affected. Second, Section 5.1 says that subtracting the null "removes that offset at every f", and Remark 1 calls the estimator "biased upward". Table 3 (Appendix B) shows that the subtraction moves the mean estimate closer to the true value at seven of the ten settings and further from it at the three highest settings of the 36-state family, where there was no upward offset to remove. With every state controllable, the true value is 2 bits, the uncalibrated estimate is 1.996 and the calibrated one 1.985. That setting is where the 0.015 bits above comes from. Third, not an error: the rank correlation of 0.98 in the abstract is the largest value that 25 runs at five tied settings can give, so it shows only that the estimates at different settings never overlap.
We have not uploaded a revised PDF.