- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies robust offline reinforcement learning when an agent uses a compressed version of its history. Its key observation is that changing the policy can change which histories occur within the same represented state, making predictions learned from logged data unreliable. COR-FQI accounts for this additional uncertainty and selects a history representation using a conservative performance estimate. The paper provides theoretical guarantees, tabular experiments on FrozenLake, and empirical evaluations with neural networks.
Critical strengths
This paper addresses a practical challenge in partially observable offline RL: retaining too little history can hide information needed for good decisions, while retaining too much makes learning harder with limited data. This trade-off is particularly relevant to robotics, autonomous systems, and industrial control, where observations or actions may be delayed and collecting new interaction data can be costly or risky. Selecting memory using conservative performance estimates could be a promising direction to learn policy that is more reliable during deployment.
Critical weaknesses
-
The writing and rigor of this paper are poor. A lot of definitions are introduced with words rather than formal mathematical formulas. For example, in row 131, the definition of is stated by words: a conditional distribution under , which is fully unspecified. This writing ambiguity makes the paper hard to read.
-
The proposed method needs reliable bounds on how much outcomes can differ between histories assigned to the same represented state, including histories absent from the dataset. Finer logging information helps, but still requires additional assumptions and coverage. The FrozenLake example uses a known delay structure, leaving applicability to less structured environments unclear.
Questions for authors
-
How can useful history-variation bounds be obtained when neither a known delay model nor a predictively sufficient finer representation is available?
-
Compared with prior literature, what is the advancement in theory and algorithm of this paper?
3 comments
(Part 2/2) Response to Reviewer 2gMY
Q1. How can useful bounds be obtained when neither a delay model nor a sufficient finer representation is available?
Section 4.2 and Appendix A.11 give four routes. Under smooth dependence (Lemma A.5), the designer states the resolution of a recorded quantity and the sensitivity of the outcome to it, and no delay is involved. Under fading memory (Lemma A.6), a hidden state that mixes gives a bound that decreases geometrically in the window length; this lemma requires bounded likelihood ratios, and the paper states this scope. With a finer label that is itself not sufficient (Proposition 4), only a residual bound is required, and the same proposition gives a lower bound that detects an insufficient representation from data. When no bound is known (Proposition A.7), the certificates for all assumed values hold on one event, and the designer can report the largest conditional variation under which a required return remains certified; the new sensitivity curve for FrozenLake in Appendix C.1 is an example.
The grid-robot study illustrates two of these routes. With the bound obtained from finer logs, the selected certificate reaches 0.709 at N = 100,000, against 0.826 with the supplied bound, for the same selected policy. The lower bound of Proposition 4 identifies both insufficient representations from data, with no false alarm at the cells where the representation is sufficient.
Q2. What is the advancement in theory and algorithm over prior literature?
The introduction now lists four contributions. The first is an impossibility result: the radius required after compression is characterized exactly, and the compressed data cannot identify it (Proposition 1). The second is calibration: the sets contain the outcome distribution of every pooled history, for all candidate representations simultaneously, although the data are dependent and not identically distributed (Theorem 3). The third is certification without a Markov state: the recursion on a represented state is a lower bound on the worst-case return, and an upper value bounds every history-dependent policy (Theorem 5). The fourth is certified selection: all certificates hold on one event, so that the representation with the largest certificate is selected from the same data without loss of confidence, with a computed upper bound on the loss relative to the best history-dependent policy (Theorem 6).
Thanks for the authors' response. However, given the revised version, I still find it hard to follow and understand the paper. I believe that the paper is not ready for acceptance at a top-tier AI conference. Therefore, I decide to maintain my score.
(Part 1/2) Response to Reviewer 2gMY
We thank the reviewer for their careful reading. The revision keeps the original statements of the main results and their numbering, and adds material around them; all changes are marked in red in the revised paper.
We have replaced the verbal definitions by formal ones. The logging weights are now defined in Section 3.1 as a regular conditional distribution of the history given the represented state and the action, under the distribution of a recorded trajectory. The dataset, the visit counts and the empirical distributions are defined by formulas in Section 4.1, the data condition of Assumption 1 is stated formally, and the visit averages in Proposition 4 are written out. The distribution of a recorded trajectory, the reset distribution and the behavior policy are introduced in Section 2.1. The proofs in the appendix are written in full sentences, and each begins with a proof sketch.
The first two statements are correct, and the paper now says so. Section 4.2 states that the bound must hold for all histories of a cell, including histories that are absent from the dataset, and that it is therefore a statement about the system and not an estimate; Proposition 1 shows that no method can obtain it from the compressed data. The same section states that calibration from finer logs still requires an exhaustive partition and valid residual bounds. A conservative bound remains safe, since it lowers the certificate without invalidating it.
On the third statement, we have made two additions. First, the sources of the bound are no longer restricted to delays: Lemma A.5 covers quantities that are recorded coarsely, and Lemma A.6 covers a hidden state that mixes, and neither uses a delay structure. Second, Appendix C.2 adds a tabular study without delay structure, a grid-robot navigation task in which two junctions look alike and a landmark seen earlier identifies the correct exit. In this study the certificates of the candidate representations cross as the data grow, and selection between two insufficient representations returns a policy that attains the full-history optimum. Without the bound, the certificates are invalid: under a history-dependent behavior policy, all 60 fits at the two largest sample sizes report a lower bound above the true return, 0.870 against 0.76 at N = 100,000.
The FrozenLake experiment also reports the effect of the inputs. Doubling the supplied bound lowers the certificate from 0.333 to 0.133, and calibration from logged labels, without the supplied bound, gives 0.343. Appendix C.1 now adds a sensitivity curve (Figure 4): the certificate of the one-command memory decreases smoothly as the assumed bound grows, from 0.403 at 0.075 to 0.333 at the supplied value 0.1 and to 0.218 at 0.15, so that a required return of 0.2 remains certified for any assumed bound up to 0.15.