- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper discusses a very interesting and novel view of exploration. A policy is a good exploration policy if the data collected using that policy is useful to predict the outcome distribution of all other policies. Their main technical contribution is that they show 1) many popular exploration methods don't maximize this notion of exploration, and 2) maximizing information gain (with a certain objective) is a reasonable surrogate objective for this notion of exploration under certain assumptions. They also introduce an interactive exploration bonus and calculate it for some very simple and small MDPs with known dynamics.
Critical strengths
-
The notion of exploration they introduce is really interesting and novel. While conceptually very similar to epistemic uncertainty, it is truly a new way to look at epistemic uncertainty that I haven't seen in teh literature before.
-
The main theorem (Thm4.1) result is interesting and definitely non-trivial. Showing a connection between information gathering and the notion of exploration through producing good counterfactuals is very interesting and novel.
-
Their analysis in sec 3.2, where they show certain information gathering objectives prefer getting less number of information that are presented to them in multiple steps than one big chunk of information presented over one step, is very interesting. I haven't seen this result before, and this raises serious concern about using some og these objectives, a concern that no prior work has introduced based on my knowledge.
Critical weaknesses
While I find the conceptual formulation of the paper very interesting and thought-provoking, I don't think this paper deserves an accept for three reasons:
-
The paper is extremely badly written. I had a horrible time understanding their results. In the Intro section, they start motivating their problem with an interesting example. The example itself is very intuitive and straightforward, but unfortunately they do a very bad job explaining it such that the example sounds too complicated, and only through reading again and again can one decode what the example is, and only then does it become so evident that they are explaining a very intuitive problem in a very unintuitive way. The rest of the paper is very badly written too. Most claims and explanations are presented without first giving enough context or using language that is unnecessarily technical and not understandable. The theorem statements are basically unreadable. If I were the author, I would consider a major edit to the writing.
-
My second big problem with the paper is that although their conceptual contribution is interesting, their objective is completely incomprehensible and they don't even make a simple attempt to present a tractable version of it. Reading their paper, I conjecture that using generative adversarial style objectives, it should be possible to obtain a tractable approximation of their method, but they didn't make any effort to obtain such algorithms, and this doesn't allow us to understand if this method is merely an interesting mental model or is actually something useful.
-
Finally, I find that some of the claims of the paper are not novel, and at the same time they don't acknowledge that these concepts have existed in the literature before; for instance, their discussion about novelty-seeking exploration methods could be summarized as the difference in aleatoric and epistemic uncertainty methods, which has been discussed in many prior RL exploration works. Their random color generator example is exactly equivalent to the popular noisy TV example in RL. Their notion of exploration is conceptually very similar to epistemic uncertainty, and they fail to mention that connection. I agree the paper presents a novel view of this kind of uncertainty, but I don't like the fact that they don't acknowledge that a big part of these concepts have been laid out by prior works. Overall, the framing of the work in the context of prior works is really weak and non-comprehensive.
Questions for authors
- Can you try a tractable approximation of the method by training three models concurrently as follows:
- A decoder model that predicts the future of a dictionary of policies
- An adversarial policy that tries to add new policies to the policy dictionary such that the decoder has a very high error on those policies
- An exploratory policy that is rewarded for collecting trajectories that reduce the encoder error
I understand the main contribution of the paper is theoretical, and it is indeed an interesting theoretical work, but some attempt to at least propose a tractable approximation of the method would significantly strengthen the work.
2 comments
Thanks for the authors' response.
One of my main concerns was about the poor presentation of the paper. Although the authors have made some changes, I still find the paper very hard to follow. For instance, starting a paragraph with the sentence "An alarm may ring once after an unknown delay or never" is genuinely a bad idea, as the reader has no context for what this means.
We study exact reward maximizers to separate these incentives from the difficulty of learning an optimal policy.
This is another example of such a hard-to-read sentence; there are many of them in the paper, unfortunately.
Your tabular experiment is interesting, but I have two concerns. First, the presentation of this experiment in the paper is very unclear, to the point that I couldn't understand it just by reading the paper. Perhaps adding a figure would help make the setup and results easier to understand.
Second, I don't agree that implementing a simpler version of this approach with neural networks would be very difficult. For example, the difficulties you mentioned could be addressed as follows:
Total variation distance between distributions over histories.
You don't need to measure this accurately. You could train a model that takes a policy index and generates trajectories of that policy, then compare these generated trajectories with ground-truth trajectories. The average distance between them could serve as a rough measure of how well the model captures the trajectory distribution. If both the policy and environment are deterministic, this comparison could be almost exact.
The criterion takes the worst case over worlds, and sampled worlds give no guarantee for unsampled ones.
In practice, you could simply measure, on average, how well a model can predict the future trajectories of other policies. It doesn't necessarily have to be a worst-case criterion.
Training compares collector and target runs in the same world, which requires resetting it.
You could train many target policies as probes that expand the frontier of exploration, alongside one collector policy whose only objective is to collect informative data. This would look somewhat like self-play.
Overall, although the authors' clarifications were helpful and the fact that they added the needed prior works to their paper is appreciated, I still don't find the presentation and results of the paper ready for publication at ICLR. I think the theoretical contribution of the paper alone, at least in the way it is presented now, isn't enough for publication, and I do believe that with substantially better presentation and with even a very simple didactic neural network-based experiment, their paper could be great.
Thank you for the careful reading and the specific suggestions. In short, we rewrote the exposition, we ran the three-model experiment you proposed (Appendix I.4), and the paper now states its connection to the noisy TV and to epistemic uncertainty.
1. The writing
We agree, and the revision rewrites the exposition. The introduction now describes the example in words before any notation and states its conclusion directly after Figure 1, and we removed the random startup bits that obscured it. Sections 2 and 3 open with concrete cases, each theorem is preceded by one sentence in plain words, and Table 1 gives one number per objective.
2. A tractable version
The score in §5 is an existence result and not a proposed training objective, which the reviewed version left too implicit. It shows that a single number can rise with every strict improvement in exploration, and Proposition H.1 shows that no algorithm can approximate such a score to any requested accuracy for every computable policy.
Two things are already available. On a finite class of worlds, each with positive prior probability, a familiar objective is aligned with the criterion, because cumulative information gain rises with every strict improvement (Theorem 4.1). And for a supplied finite world model with a fixed list of targets and a fixed collection budget, §6 computes finite versions of the score numerically.
We also tried your three-model proposal, in a tabular setting with two hidden bits (Table 3). We trained a collector, one decoder per target policy and a target generator from sampled episodes, and then audited the collector against exact errors from linear programming. In two of three environments its worst error over all 27 targets, averaged over six seeds, is within 0.0002 of the optimum under the same minimum action probability, and in the third, which has noisy readings, it reaches 0.094 against 0.078. It also keeps its first-action probability of requesting a fair coin near the minimum allowed, so it does not chase noise. Three caveats apply: the fitted decoders stayed weaker than the optimal ones used in the audit, the learned generator added little over keeping the hardest of four random candidates, and training used labelled reset episodes from every world, so the pilot shows that your loop is well posed but not yet that it is practical.
The version with neural networks that we have in mind keeps your three parts: a recurrent collector; a sequence model as decoder, which reads the collector's history and predicts each next target observation; and a proposer of target policies, rewarded for the decoder's excess error over a second decoder that is told the world, a baseline meant to remove the reward for noise. We have not implemented it, and we think it is a separate paper because of four obstacles. The loss is a total variation distance between distributions over histories, which is hard to estimate from samples once histories are long. The criterion takes the worst case over worlds, and sampled worlds give no guarantee for unsampled ones. Training compares collector and target runs in the same world, which requires resetting it. And the space of target policies grows very quickly, since §6 already has 32,768 four-step targets.
3. Prior work
You are right, and we should have said so. The random color is a noisy TV (Burda et al., 2019), and a reward for it is a reward for aleatoric uncertainty (Mavor-Parker et al., 2022), which the introduction now states. The Shannon, Square and count results in §3.1 are therefore versions of that known failure. Three other results are not: empowerment ignores the color, yet each of its maximizers keeps the display with positive probability because it is rewarded for controlling the digit; the four-bit example in §3.2 has no noise, yet Hellinger and Absolute still prefer three bits one at a time to four at once; and information gain, which counts only epistemic uncertainty, still ties inspection with play under the complete return (§3.3).
On epistemic uncertainty, §7 now says that expected information gain measures the epistemic uncertainty removed on average over worlds under a prior, whereas our criterion requires simulation in every world and chooses no prior. Theorem 4.1 relates the two on finite classes, and the alarm shows that they can separate on an infinite one: after a long silence the average uncertainty is nearly zero, but at every time some world still has its alarm ahead.