- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an evaluation metric to measure behavioral similarity in the specific sense of: the similarity/difference between two states is based on how similar/different the future states are when the same sequence of actions is executed from each state (based on Figure 1). The paper's claim is that two states should have similar or dissimilar representations if the future states under this type of rollout are similar/dissimilar. The evaluation metric and algorithm, QueryPrint, analyzes various representation learning algorithms such bisimulation-based algorithms and various representations from pre-trained encoders such as DINO and JEPA. The empirical evaluations focus on pixel-based Atari games.
Critical strengths
- Diagnostic efforts. I appreciate the effort to diagnose a wide collection of existing algorithms rather than trying to propose another algorithm in the zoo of algorithms. Its important for the community to step back and understand how our current algorithms may be equipped or not to tackle different questions based on the properties they exhibit.
Critical weaknesses
- Poor readability. Unfortunately, its very difficult to parse this paper, even after spending several hours. For example, majority of the terms in the abstract are atypical and I did not understand their meaning until much later in the paper. In the empirical section, it would also be useful to bring back the defined notation from Section 4 and say how each value is set rather than saying what the hyperparameters/empirical setup is in words. It was also challenging to understand Section 4 and the intuition behind the metrics.
- Unclear if this behavior similarity notion actually matters. While it is nice that the effort is to diagnose existing algorithms, I would have expected the flow of the paper to be: 1) show that this similarity actually impedes performance, 2) present an evaluation metric that measures divergence from this ideal notion of similarity, 3) provide an algorithm to fix the issue, and 4) show how the new representations correlate better with the proposed metric. Figure 5 is a step in the direction I have in mind where there is a connection between the metric and agent behavior, but this effort is generally sparse in the paper.
Questions for authors
- Maybe I am misunderstanding but is there an ideal value for entries in Table 1?
- The empirical section states that 600 states were used for the analysis but this seems far too little (even noted in the paper that it is 4% of the typical amount of experience) to draw conclusions. How were these number of states chosen? Additionally, the learned objectives were on the 600 states, that seems far too little to actually learn a reliable representation. It seems obvious that they would be poor representations.
- The evaluation with 3 seeds (and less data of 600 states) makes it hard to read the results confidently. Regarding seeds, I strongly recommend the readers look at Empirical Design in Reinforcement Learning (Patterson et al. 2024). Was there a sense of why 3 was enough?
2 comments
Thank you for the response and clarifying few of my concerns.
While some questions have been answered, I still think the overall issues of the paper remain: 1) the lack of measurement of actual return with the said representations; 2) the poor readability still makes it very difficult to understand the core message of the paper and contextualize results; and 3) I think the changes and improvements that need to be made (those promised in the current response by the authors and additional changes (my 1 and 2)), suggest that there is lot of work yet to be done, so I will still stick with my score.
Response to Reviewer kdRf
We thank the reviewer for the feedback. We address each point below.
W1. Readability
We will start Section 4 with a formal definition of the distance, given in our response to Reviewer fA4g. We will also write the setup of Section 5.1 in the notation of Section 4. There are six query policies, three for training and three for testing, with N=600 states per game, R=4 repeats, and a horizon of H=600 steps. In the abstract and the introduction we will define query policy and fingerprint where they first appear.
W2. Relevance of this notion of similarity
We agree that the paper does not measure the return of an agent trained on each representation. We test the link to agent behavior with a policy outside the measurement. The measured distance predicts where an independently trained PPO agent loses a life. Across the 43 distances in Figure 5, how well a distance agrees with the measured geometry predicts how well it anticipates this outcome, with a correlation of 0.93. In Appendix J the correlation is between 0.87 and 0.94 at every PPO checkpoint, and it is still 0.93 without our encoder. We did not use the return because, as noted in Section 1, it also depends on the policy learner and can leave the geometry of the representation unexamined (Luo et al., 2025).
Q1. Ideal value in Table 1
A score of 1 means that a distance orders all state pairs as the measured target does. The footnote of the table gives two reference values. The split-half reliability of 0.941 is the score of an independent repetition of the measurement. The agreement of 0.896 between training and test queries is the score of the target that the encoder and the heads are fit to. The encoder reaches 0.839, which is 94% of the latter.
Q2. Number of states
The 4% is the cost of the measurement in emulator steps relative to a 200M-frame run. The measurement takes 600 states times 6 queries times 4 repeats times 600 steps, which is 8.64M steps. Four results in the paper bear on whether 600 states are too few.
Q3. Number of seeds
Each row of Table 1 aggregates 165 runs, three seeds on each of 55 games. Following Agarwal et al. (2021), we bootstrap over games and pair differences by game. Most of the variation is between games. For our encoder, the standard deviation over seeds within a game has a median of 0.005, compared with 0.069 between games. For the CURL head the two values are 0.016 and 0.173, and for the DINOv3 head they are 0.007 and 0.108, so more seeds would change the results little. We will report the seed-level variation in the appendix, as recommended by Patterson et al. (2024).
Please let us know if there are further questions.