acceptodds
Under review as a conference paper at ICLR 2027

QueryPrint: Measuring Behavioral Similarity with Counterfactual Futures

Abstract

Representation learning transforms observations into features for prediction and control. For control, a useful notion of state similarity asks whether the same policies lead to similar futures. Existing behavioral and predictive methods learn state relationships from trajectories, which show one future of each state and rarely the futures of different states under the same policies. We introduce QueryPrint, which restores saved states in resettable environments, executes fixed query policies, and summarizes the resulting counterfactual futures into fingerprints. Distances between fingerprints define a query-relative behavioral target, which we test for action dependence, reliability, and non-degeneracy and use to supervise a compact encoder. Across 55 Atari games, this encoder captures the measured geometry in its native distances and generalizes to unseen states and query policies. In contrast, the 9 learned objectives and 12 frozen pretrained representations we evaluate show little of this geometry in their native distances. Heads trained on the same target recover much of it from frozen features and reach the encoder's level at matched capacity. The measured distance also predicts differences in lives lost under independently trained PPO agents, while a learned world model recovers the geometry only partially. Replayed futures provide a directly measured target for learning and evaluating behavioral geometry, and show that information recoverable from features need not appear in native distances.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, bisimulation metrics, representation learning, behavioral similarity, pretrained visual representations, Atari, evaluation, world models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kdRfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an evaluation metric to measure behavioral similarity in the specific sense of: the similarity/difference between two states is based on how similar/different the future states are when the same sequence of actions is executed from each state (based on Figure 1). The paper's claim is that two states should have similar or dissimilar representations if the future states under this type of rollout are similar/dissimilar. The evaluation metric and algorithm, QueryPrint, analyzes various representation learning algorithms such bisimulation-based algorithms and various representations from pre-trained encoders such as DINO and JEPA. The empirical evaluations focus on pixel-based Atari games.

Critical strengths
  1. Diagnostic efforts. I appreciate the effort to diagnose a wide collection of existing algorithms rather than trying to propose another algorithm in the zoo of algorithms. Its important for the community to step back and understand how our current algorithms may be equipped or not to tackle different questions based on the properties they exhibit.
Critical weaknesses
  1. Poor readability. Unfortunately, its very difficult to parse this paper, even after spending several hours. For example, majority of the terms in the abstract are atypical and I did not understand their meaning until much later in the paper. In the empirical section, it would also be useful to bring back the defined notation from Section 4 and say how each value is set rather than saying what the hyperparameters/empirical setup is in words. It was also challenging to understand Section 4 and the intuition behind the metrics.
  2. Unclear if this behavior similarity notion actually matters. While it is nice that the effort is to diagnose existing algorithms, I would have expected the flow of the paper to be: 1) show that this similarity actually impedes performance, 2) present an evaluation metric that measures divergence from this ideal notion of similarity, 3) provide an algorithm to fix the issue, and 4) show how the new representations correlate better with the proposed metric. Figure 5 is a step in the direction I have in mind where there is a connection between the metric and agent behavior, but this effort is generally sparse in the paper.
Questions for authors
  1. Maybe I am misunderstanding but is there an ideal value for entries in Table 1?
  2. The empirical section states that 600 states were used for the analysis but this seems far too little (even noted in the paper that it is 4% of the typical amount of experience) to draw conclusions. How were these number of states chosen? Additionally, the learned objectives were on the 600 states, that seems far too little to actually learn a reliable representation. It seems obvious that they would be poor representations.
  3. The evaluation with 3 seeds (and less data of 600 states) makes it hard to read the results confidently. Regarding seeds, I strongly recommend the readers look at Empirical Design in Reinforcement Learning (Patterson et al. 2024). Was there a sense of why 3 was enough?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kdRf

We thank the reviewer for the feedback. We address each point below.

W1. Readability

We will start Section 4 with a formal definition of the distance, given in our response to Reviewer fA4g. We will also write the setup of Section 5.1 in the notation of Section 4. There are six query policies, three for training and three for testing, with N=600N = 600 states per game, R=4R = 4 repeats, and a horizon of H=600H = 600 steps. In the abstract and the introduction we will define query policy and fingerprint where they first appear.

W2. Relevance of this notion of similarity

We agree that the paper does not measure the return of an agent trained on each representation. We test the link to agent behavior with a policy outside the measurement. The measured distance predicts where an independently trained PPO agent loses a life. Across the 43 distances in Figure 5, how well a distance agrees with the measured geometry predicts how well it anticipates this outcome, with a correlation of 0.93. In Appendix J the correlation is between 0.87 and 0.94 at every PPO checkpoint, and it is still 0.93 without our encoder. We did not use the return because, as noted in Section 1, it also depends on the policy learner and can leave the geometry of the representation unexamined (Luo et al., 2025).

Q1. Ideal value in Table 1

A score of 1 means that a distance orders all state pairs as the measured target does. The footnote of the table gives two reference values. The split-half reliability of 0.941 is the score of an independent repetition of the measurement. The agreement of 0.896 between training and test queries is the score of the target that the encoder and the heads are fit to. The encoder reaches 0.839, which is 94% of the latter.

Q2. Number of states

The 4% is the cost of the measurement in emulator steps relative to a 200M-frame run. The measurement takes 600 states times 6 queries times 4 repeats times 600 steps, which is 8.64M steps. Four results in the paper bear on whether 600 states are too few.

  • All learned objectives use the same 600 states, network, and update budget as our encoder, so only the source of supervision differs.
  • The objectives do learn from these states. Through the behavioral head in Table 1, CURL reaches 0.735 and ST-DIM 0.632, against 0.207 for a random encoder.
  • More data does not change the result. In Table 14, with four times as many states on five games, our encoder scores 0.885 at 600 states and 0.875 at 2,400, and CURL scores 0.656 and 0.652. In Table 13, no objective exceeds 0.694 with up to 8,000 updates on 20 games.
  • The 12 frozen pretrained models are not trained on these states at all, and their native distances are just as low, at most 0.292. The amount of training data does not explain the low native scores.

Q3. Number of seeds

Each row of Table 1 aggregates 165 runs, three seeds on each of 55 games. Following Agarwal et al. (2021), we bootstrap over games and pair differences by game. Most of the variation is between games. For our encoder, the standard deviation over seeds within a game has a median of 0.005, compared with 0.069 between games. For the CURL head the two values are 0.016 and 0.173, and for the DINOv3 head they are 0.007 and 0.108, so more seeds would change the results little. We will report the seed-level variation in the appendix, as recommended by Patterson et al. (2024).

Please let us know if there are further questions.

Thank you for the response and clarifying few of my concerns.

While some questions have been answered, I still think the overall issues of the paper remain: 1) the lack of measurement of actual return with the said representations; 2) the poor readability still makes it very difficult to understand the core message of the paper and contextualize results; and 3) I think the changes and improvements that need to be made (those promised in the current response by the authors and additional changes (my 1 and 2)), suggest that there is lot of work yet to be done, so I will still stick with my score.

Reviewer_4hp7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QueryPrint, which measures state similarity by restoring saved emulator states and running the same query policies from each state. It summarizes repeated rollouts into fingerprints and uses selected fingerprint distances to evaluate and train representations. The paper evaluates QueryPrint on 55 Atari games, with further tests on new states, expert queries, and outcomes under independent PPO policies. The results show that QueryPrint’s encoder achieves high agreement with the proposed distance, and that supervised heads improve agreement for many baseline representations.

Critical strengths

The paper shows that supervised heads can recover information that native feature distances do not reflect, helping clarify how representation quality should be assessed. Its evaluation across 55 Atari games supports this finding, while the independent PPO results provide useful external validation of the measured distance.

Critical weaknesses

1. The novelty and benefits of the method are not sufficiently established.

Comparing states through their future behavior under common actions or policies is well studied. Relevant work missing from the discussion includes policy-independent behavioral metrics (Liao et al., AAAI 2023), random action-conditioned predictions (Zheng et al., NeurIPS 2021), and distributional successor measures (Wiltzer et al., ICML 2024). The paper should clarify what its fingerprint construction and measurement selection add beyond these approaches.

The experiments do not clearly establish the benefits of these design choices. Section 5.4 argues that each measurement component matters, but the ablations in Table 22 are evaluated against the full QueryPrint target. Lower agreement with that target does not show that an alternative is less useful. The outcome-only controls in Table 12 have the same limitation. The independent PPO results provide useful external validation, but do not establish whether the proposed statistics and selection procedure improve on simpler alternatives. Comparing outcome-only, mean-only, and fixed-measurement targets under the same PPO evaluation and rollout budget would help establish the value of the additional measurement steps.

References:

  • Liao et al., Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning, AAAI 2023.

  • Zheng et al., Learning State Representations from Random Deep Action-conditional Predictions, NeurIPS 2021

  • Wiltzer et al., A Distributional Analogue to the Successor Representation, ICML 2024.

2. The validity tests do not establish reliable action-dependent information in the final distance.

The tests in Section 4.2 assess different parts of the fingerprint. The shuffle test examines only variation across queries, while reliability and the final distance use the full fingerprint. A small query-specific component can therefore pass the shuffle test even when shared variation dominates the final distance.

Sampling noise creates a further problem. Even if all query policies have the same future distribution at each state, their estimated fingerprints can differ because of noise. These differences can produce low shuffle retention, while stable shared information keeps the full distance reliable. The independent-half check in Appendix G does not resolve this: low agreement after shuffling can also occur when the query-specific estimates already disagree across independent halves.

Shared variation can be meaningful, but its reliability does not establish that query-specific differences are reliable or meaningfully shape the final distance. This leaves the claimed action dependence and exclusion of task-irrelevant variation insufficiently supported. The authors should report unshuffled cross-half agreement for the query-specific distances, compare it with shuffled agreement, and compare the full and shared-only distances.

Questions for authors
  1. Tables 12 and 22 evaluate simpler measurements mainly against the full QueryPrint target. Using the same recorded rollouts, how do the full, outcome-only, lives-only, mean-only, and fixed-measurement distances compare in the independent PPO evaluation?

  2. The shuffle test uses the query-specific component, while reliability is measured using the full fingerprint. How well do the query-specific distances agree between independent halves of the rollouts, before and after query-label shuffling?

  3. How much of the final QueryPrint distance is explained by information shared across queries? How do the shared-only and query-specific-only distances perform in the independent PPO evaluation?

  4. In the three-query ablation (Appendix K), only one training query remains, making the query-specific component in Equation (3) zero. How is measurement selection performed in this setting?

  5. Appendix L constructs imagined fingerprints from DINOv3 feature changes, while the measured fingerprints use projected raw-frame changes. Do the imagined fingerprints include outcome statistics? How do real and imagined rollouts compare when using the same DINOv3-based fingerprint?

  6. Appendix M fits the PSE metric using a PPO reference policy, but the stored transitions come from procedural policies. How are the PPO-policy successor transitions obtained or accounted for?

  7. Table 3 describes CMD and TMD as learning from trajectories, while Section 5.1 states that the baselines use stored transitions from 600 states. What trajectory information and temporal gaps are actually used to train these methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4hp7 (1/2)

We thank the reviewer for the careful review and the specific questions. We ran the requested analyses on the recorded rollouts of the 18 test games, with no new emulator steps. The recomputed distances match the stored target of every game and reproduce the PPO score of 0.265 in the paper.

W2, Q2. Reliability of the query-specific distances

We split the four repeats into two independent halves in each of the three possible ways and computed every distance within each half. The table gives the Spearman correlation between the two halves on the training queries, as the median over games.

DistanceAgreement between halves
Full fingerprint0.921
Shared part only0.930
Query-specific part0.598
Query-specific part, query labels shuffled in one half0.174

The unshuffled agreement is higher than the shuffled one in all 18 games, with a paired difference of 0.41 and a 95% CI of [0.29, 0.51]. On the test queries the two values are 0.469 and 0.242. If all queries had the same future distribution at a state, shuffling would only exchange independent draws of the same quantity, and the last two rows would be equal. They differ in every game, so sampling noise cannot explain the query-specific differences. We will add this comparison to Appendix G.

Q3. Shared and query-specific parts of the final distance

Most of the final distance is shared across the queries. On the test queries the shared part accounts for 72% of the variance between states. A distance built from it alone agrees with the final distance at a median of 0.985, and one built from the query-specific parts alone agrees at 0.544. The shared part is the average future of a state under the query set, so it is still a counterfactual quantity under matched policies, and Section 4.2 defines the distance over both parts. Under the PPO evaluation the shared-only distance scores about the same as the full one on lives lost, 0.274 against 0.265, and the query-specific-only distance scores 0.114, in line with its lower reliability.

In Section 4.2 the shuffle test screens and ranks the candidate measurements, for instance by removing those dominated by a timer, and reliability describes the distance that is used as the target. We will state these two roles separately there and report the three distances in Appendix G.

W1, Q1. Simpler targets under the PPO evaluation

Following the reviewer's suggestion, we compared the simpler targets under the PPO evaluation of Appendix J, using the same rollouts and test queries. The table gives the score on lives lost, the primary outcome, for the final policy. The fixed measurements use a horizon of 600 steps and a discount of 0.99 in every game.

DistanceLives lost
Full, as in the paper0.265
Mean statistic only0.261
Fixed memory measurement0.225
Fixed frame measurement0.213

Per-game selection scores higher than both fixed measurements at all four PPO checkpoints, significantly so at the 5, 20, and 50 percent checkpoints. The repeat statistics beyond the mean do not change the score.

The outcome-only and lives-only distances score at least as high as the full distance on lives lost. This is expected, since the evaluation scores the same kind of quantity that these statistics record under the queries. They are not usable as targets, however. The lives statistics carry almost no information about reward, 0.03 against 0.15 for the full distance, and are undefined in 4 of the 18 games. The outcome statistics give zero distance to more than 65% of the state pairs in 5 of the 18 games, and their agreement with the full target is 0.50 over the 55 games in Table 12.

Table 22 reports how much the target changes when one component is changed. We will say so in Section 5.4 and add this comparison.

Continued in the next comment.

AuthorsOpenReview ↗

Response to Reviewer 4hp7 (2/2)

Main result under different targets

We also rescored the representations against variants of the target on the 18 test games.

TargetHighest native distance among baselinesQueryPrint encoder
Full0.3930.838
Shared part only0.3930.850
Mean statistic only0.3950.852
Fixed memory measurement0.3210.595
Outcome statistics only0.2540.494

The baselines are 8 learned objectives and 12 frozen models, with PSE left out as in Appendix J. The highest native distance stays below 0.40 under every variant. The encoder was trained on the full target, which is why it scores lower on the last two rows. The main result, that native distances of current representations do not show this geometry, holds for each way of building the target.

W1. Related work

We thank the reviewer for the three references and will add them. Liao et al. (2023) propose a policy-independent behavioral metric, Zheng et al. (2021) use random action-conditional predictions as auxiliary tasks, and Wiltzer et al. (2024) learn a distributional successor measure. All three are learning objectives trained on the agent's experience. Our notion of similarity is in the same family. The difference is that we replay the futures of different states under the same policies from saved states. The distance is then measured with a known reliability and can score any representation, including ones trained with these objectives. The closest experiment in our paper to Zheng et al. is the future-prediction encoder in Appendix K. It predicts the same futures without a geometric term and reaches 0.413 with its native distance and 0.815 through a head.

Q4. Three-query ablation

No selection is run in this variant. It keeps the three measurements that the full procedure selected for the game and recomputes the fingerprint from a single training query, chosen per game by a fixed hash. The target is the standardized full fingerprint of that query, so Equation 3 is not used. Scoring uses the test-query distance of the full target. We will add this description to Appendix K.

Q5. Imagined fingerprints

No, the imagined fingerprints contain no outcome statistics. They are built only from the change of the predicted DINOv3 patch features. Appendix L checks whether this mismatch matters by scoring them against the frame-change measurement of each game, the signal family the model observes, and the score is the same 0.345. For the comparison the reviewer asks about, we replayed the same test queries in the emulator and built the same DINOv3-based fingerprint from the real frames. At the 600-step horizon it reaches 0.400, against 0.345 for the imagined rollouts. This difference is small compared with the gap to the encoder's 0.839, so prediction error explains little of it. We will add this row to Appendix L.

Q6. PSE

In the offline setting the only transitions available are the stored ones, taken by the procedural policies that collected the states. The PPO policy provides the action distributions of the total-variation term, and the successor pairs are the stored successors. The fitted quantity is the policy similarity metric of the PPO action distributions under the stored transitions. All learned objectives in the comparison are in the same situation, because none of them interacts with the emulator. Appendix M will say this explicitly. PSE is left out of the PPO outcome evaluation because its metric is fitted from that policy.

Q7. CMD and TMD

Both are trained on the 600 stored states and their one-step transitions. The future state of a state is another stored state of the same episode, drawn at a geometric offset with parameter 0.01 in the ordered list of the stored states of that episode. Stored states of one episode are multiples of 80 emulator steps apart, and the episodes were played by the six procedural policies. Table 3 lists the data that the original methods assume, and Section 5.1 describes the data we use. We will make the two consistent and report the temporal gaps.

Please let us know if anything remains unclear.

Thank you for the response and additional experiments. The cross-half results substantially address my concern about sampling noise. The comparison with fixed measurements also provides some support for the selection procedure.

My main concern remains the benefit of the full measurement design. Mean-only and shared-only distances perform similarly to the full distance in the PPO evaluation of lives lost, and the shared-only distance agrees with it at 0.985. The added value of the statistics beyond the mean and the query-specific component therefore remains unclear.

I agree that shared variation can be meaningful. However, showing that reliable query-specific information exists does not establish that the selection procedure excludes distances dominated by task-irrelevant shared variation. The real-rollout world-model control also suggests that much of the reported gap reflects the fingerprint construction rather than prediction error.

The available PDF remains unchanged. The original claims about the validity tests and world-model comparison should be revised to reflect these findings.

I appreciate that the response resolves several concerns and strengthens the empirical results. However, I am still not convinced that the proposed measurement design offers a sufficiently clear methodological contribution, so I maintain my rating.

Reviewer_fA4gOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates the learned representation of several representation learning algorithm such as, V-JEPA, TMD, DINOv2. The paper proposes a new measure of behavioral similarity that leverages the access to a simulator and the ability to reset to a desired state to roll multiple policy in the environment, and measures changes in state simulator, game and pixel features averaged over multiple short horizons of lenght H. The learned fingerprints are then used for evaluating representation learning algorithm and additionally to train a new encoder from leveraging this priviliged training dataset.

Critical strengths
  • The idea is interesting. While it is not entirely novel, as several prior works have highlighted the importance of representations that capture environment dynamics for control, I believe this is the first work to extensively evaluate multiple representation learning algorithms based on their representation quality for control.
Critical weaknesses
  • I found the paper difficult to read, in part because it lacks a clear formal definition of the objective underlying the proposed behavioral measure.

  • The lack of a formal objective makes it difficult to follow the paper and assess whether the family or subset of policies selected for the measure is appropriate.

  • The paper is also difficult to follow because some key concepts are introduced without sufficient explanation, such as the outcome signal yy. Important details, such as the behavioral policies used, are also deferred to the appendix.

  • The paper first introduce the general formula for the fingerprints and then spend multiple subsection to defining tests for determining whether a fingerprint can serve as a target. Again, this seems to stem from the lack of a clear formal definition of the objective of the proposed measure.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fA4g

We thank the reviewer for the feedback. The four weaknesses share one request, a formal statement of what the measure estimates. The paper gives it in words in Section 3 and through Equations 1 and 2, and we agree that it should be written out formally. We do so below and will place it at the start of Section 4.

Objective

Fix a set of query policies Π\Pi, a horizon HH, and a discount γ\gamma. For a state ss and a policy π∈Π\pi \in \Pi, run π\pi from ss for HH steps and accumulate the discounted change signal and the outcome signals as in Equation 1. Let Zπ(s)Z_\pi(s) denote the resulting random vector. Its distribution is determined by the transition kernel and π\pi. Let ϕ\phi collect the mean, standard deviation, and 10th and 90th percentiles of each coordinate of a distribution. The behavioral distance is

dΠ(s,s′)=1∣Π∣∑π∈Π∥ϕ(Law Zπ(s))−ϕ(Law Zπ(s′))∥1,d_\Pi(s, s') = \frac{1}{|\Pi|} \sum_{\pi \in \Pi} \big\| \phi(\mathrm{Law}\, Z_\pi(s)) - \phi(\mathrm{Law}\, Z_\pi(s')) \big\|_1,

with each coordinate standardized over states. It is a pseudometric on states, and it is zero when the two states have the same distribution of futures under every query policy, up to the statistics in ϕ\phi. With reward as the signal and the mean as the only statistic, each term becomes ∣VHπ(s)−VHπ(s′)∣|V^\pi_H(s) - V^\pi_H(s')|. This is the HH-step analogue of the value difference bounded by π\pi-bisimulation metrics in Theorem 3 of Castro (2020).

Estimator and tests

The fingerprint in Equation 2 is the plug-in estimate of ϕ(Law Zπ(s))\phi(\mathrm{Law}\, Z_\pi(s)) from RR replayed rollouts, and the measured distance is the corresponding estimate of dΠd_\Pi. The tests in Section 4.2 are properties of this estimator. Reliability is the agreement of the estimate between independent sets of rollouts, and the shuffle test checks whether the terms differ across the policies in Π\Pi. The tests are needed because HH, γ\gamma, and the signal are free choices, and some choices give a distance that is reproducible but unrelated to behavior, such as one dominated by a timer.

Choice of the policy set

The distance dΠd_\Pi is defined relative to Π\Pi, in the same way that a bisimulation metric is defined relative to a reward. The paper checks how much the distance depends on Π\Pi in three places. In Table 1, the distance under three separate test policies agrees with the distance under the training policies at 0.896. In Appendix J, a query set made of PPO checkpoints agrees with the procedural one at 0.724 on six games. In Section 5.4 and Figure 5, the measured distance predicts where an independently trained PPO agent loses a life.

Outcome signals and query policies

The outcome signals are the sign of the reward, termination, the number of lives lost, and whether the rollout reached the horizon. The six query policies are listed in Table 2 in Appendix A. We will move both into Section 4.1, where they are first used.

The weaknesses listed in the review concern the presentation. If the reviewer has concerns about the experiments or the conclusions, we would be glad to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.