acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Dynamics-Aware Experience Retrieval for Policy Transfer

Abstract

Changes in actuator strength, mass, or friction can make similar states require different controllers. We propose counterfactual dynamics-aware experience retrieval, which compares queries and stored experiences through their responses to shared action sequences. Reference-relative fingerprints capture response direction and magnitude; averaging their distances over each policy's memories selects a controller from a fixed library. Across four DeepMind Control tasks with six policy-training seeds per task, simulator-generated fingerprints reduce mean policy-selection regret by 49-81% relative to random selection and outperform both tested state-similarity baselines in all 24 runs. Same-probe system identification and utility regression remain competitive, with different methods leading on different tasks. With learned visual world models, validation retrieval regret selects models that improve on random selection on Hopper and Walker, whereas held-out rollout loss selects worse-than-random retrievers. These comparisons use separate query draws from the same dynamics grid. The results motivate evaluating world models through the policy choices they support.

Primary area:
reinforcement learning
Keywords:
Policy Transfer, Experience Retrieval, Counterfactual Prediction, World Models, Dynamics Shift, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VQJDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a counterfactual dynamics-aware experience retrieval method for policy transfer. The method applies shared action sequences to query states and stored experiences, constructs reference-relative dynamics fingerprints from predicted responses, and selects a controller from a fixed policy library based on fingerprint similarity. Experiments on four DeepMind Control tasks show improvements over state-similarity-based retrieval methods. However, the theoretical justification for the proposed similarity measure is insufficient, and the experiments do not convincingly establish the reliability of the method under unseen dynamics.

Critical strengths
  1. The paper addresses a relevant policy transfer problem by considering action-response information rather than relying solely on state similarity.
  2. The method selects controllers from an existing policy library without retraining them, which is useful for policy reuse.
  3. The experiments include comparisons with system identification, utility regression, and state-based retrieval, as well as an evaluation using learned visual world models.
Critical weaknesses
  1. The central assumption lacks theoretical justification, and the connection between dynamics similarity and policy performance remains unclear. The method assumes that states or environments exhibiting similar responses to the same action sequences should favor similar control policies. However, Section 4 only defines the response fingerprints and their similarity measure, without establishing a relationship between fingerprint distance and policy performance. Two environments may respond similarly to a limited set of probe actions but behave differently under the state-action distributions induced by the deployed policies. The authors should clarify under what conditions similarity under finite action probes can support reliable policy transfer, or provide a theoretical connection between fingerprint distance and policy performance loss.
  2. The similarity measure in Eq. 4 has a questionable mathematical formulation. The proposed distance combines directional similarity with a logarithmic response-magnitude ratio: [\alpha\log{\left(\frac{|C_q|_2}{|C_m|_2}+\epsilon\right)}.] However, this term is not a symmetric measure of magnitude difference. Ignoring the small ϵ\epsilon, it can be decomposed into the log norm of the query fingerprint minus that of the memory fingerprint. For a fixed query, the first term is constant, meaning that the formulation tends to favor memories with larger response magnitudes rather than those whose magnitudes are closer to the query. Moreover, the resulting distance can be negative and is not symmetric. The authors should justify this formulation and compare it with more appropriate alternatives, such as the absolute logarithmic magnitude difference. Otherwise, the claimed joint matching of response direction and magnitude is not sufficiently supported.
  3. The effectiveness of the counterfactual dynamics representation is not adequately established. Section 4.1 constructs dynamics fingerprints using eight fixed action sequences, each with a horizon of 25 steps. However, the paper does not explain why these probes are sufficient to distinguish different dynamics regimes or analyze cases where distinct dynamics parameters produce similar fingerprints. Table 2 only compares different numbers of probes, simultaneously changing the amount of response information and the computational budget. The authors should investigate the identifiability of the fingerprints and the effects of probe design, prediction horizon, and model errors on dynamics discrimination. The current experiments do not establish that the proposed fingerprints consistently capture dynamics differences relevant to policy selection.
  4. The experimental protocol provides limited evidence of generalization to unseen dynamics. In Section 5.2, world-model selection uses validation retrieval regret computed from policy returns in both source and held-out dynamics regimes. Although the final evaluation uses different query samples, the validation and test sets share the same dynamics parameter grid. Therefore, the results mainly demonstrate model selection within a known collection of dynamics regimes rather than generalization to genuinely unseen dynamics. The authors should evaluate the method on dynamics parameter configurations entirely excluded from model selection and avoid using target-regime policy returns during validation.
Questions for authors

The idea of using action-response information for experience retrieval is practically relevant. However, the proposed method largely relies on manually designed dynamics fingerprints and similarity matching, without sufficient theoretical justification. In particular, the magnitude term in Eq. 4 raises a clear mathematical concern, while the learned world-model results reveal substantial limitations when true dynamics information is unavailable. Furthermore, the model-selection protocol uses validation returns from target dynamics regimes, limiting the evidence for generalization to unseen environments.Overall, the current theoretical and experimental evidence is insufficient to support the central claim that the proposed method enables reliable policy transfer under unseen dynamics.

1 comment
AuthorsOpenReview ↗

Response to VQJD: distance definition and unseen dynamics

Thank you for examining the distance definition and evaluation protocol closely. We have revised Eq. 4, added a conditional transfer analysis, and evaluated learned retrieval on a separate dynamics grid.

  1. The magnitude term in Eq. 4.

You are right that a signed log-ratio would favor large memory norms. Our archived local source and implementation include an absolute value. The revision makes the symmetry explicit:

alpha * |log(||C_q||_2 + epsilon) - log(||C_m||_2 + epsilon)|.

Section 4.2 defines the zero-vector convention and calls the result a dissimilarity. This numerical clarification leaves all policy choices unchanged across the 24 archived task/seed runs. The learned-selector gains come from direct response prediction and source calibration; Appendix F.1 specifies the calibrated implementation's norm convention.

  1. When do finite probes support transfer?

Appendix E derives a sufficient condition for linear dynamics x'=Ax+Bu with sufficiently excited probes. Fingerprint discrepancy then bounds parameter discrepancy; with common Lipschitz rewards/policies and bounded visited states/actions, error propagation gives a return bound delta_T. The regret of transferring the policy associated with memory m is at most:

2 delta_T + [max_p J(m,p) - J(m,p(m))].

Prediction errors and initial-state differences enter delta_T. The second term captures how well the associated policy performs at the source memory. Applying the result requires the stated assumptions, which have not been verified for the nonlinear DMC experiments. Extending the result to group averaging also requires control of the memories in the selected group.

We give an explicit identifiability failure: for x'=x+u+b, x_0=0, and reward r(x)=x, reference-relative state and mean-reward contrasts are independent of b. Yet policies pi_+(x)=tanh(x) and pi_-(x)=-tanh(x) have three-step returns 3b+tanh(b) and 3b-tanh(b); their ranking reverses with the sign of b. Retaining the reference response resolves this example. Its usefulness more broadly is an empirical question.

For probe design, Table 11 compares (K,H)=(2,20),(4,10),(8,5) at KH=40, with state-only fingerprints and unchanged probe frequencies. The best allocation varies across tasks. Appendix G adds rollout, decoding, and context diagnostics to examine prediction error.

  1. Evaluation on dynamics excluded from selection.

The original validation and test samples share a dynamics grid, as you noted. We now distinguish that comparison from grid R, which contains six new parameter tuples per task. The methods compared in Table 5 and their predictions were frozen before R policy returns were collected. Distance calibration uses 144 source-query return vectors. Earlier grids informed backbone choice and method development; R was excluded from every selection decision (Appendix F).

Table 5 reports regret reductions from 15.302 to 12.540 on Walker and from 8.466 to 4.727 on Hopper, or 18.0% and 44.2%, relative to the frozen direct-response selector. Hopper's revision includes both positive weights and a log-magnitude term; Appendix G gives the unit-weight comparison. Source-fixed and utility controls, using the same return labels, achieve lower regret at 10.060 and 3.755. Each task has one frozen backbone and library, evaluated over three query draws. The revised claims reflect both the improvement on new configurations and the remaining advantage of these baselines.

Reviewer_A7xqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies policy library selection under hidden dynamics changes. When physical parameters such as actuator strength, mass, and friction change, retrieving from memory and selecting a controller based only on current-state similarity fails. The authors propose counterfactual dynamics-aware retrieval. A shared action probe sequence is used to roll out the dynamics model from both the query state and memory states, constructing response fingerprints relative to a reference sequence. Query and memory are then compared using a cosine distance, and a controller is selected by averaging distances within each policy group. The method is evaluated on four DMC tasks.

Critical strengths

The problem setting is interesting and has clear value and research significance.

Critical weaknesses
  1. The writing is not smooth enough and is somewhat difficult to read. The overall layout also needs improvement.

  2. The CF-state method uses the real simulator to generate probe responses at query time, but in real-world settings the true simulator is unavailable; what can actually be executed is CF-WM. In Figure 2, CF-ng outperforms B3-sysid on only one of the four tasks, namely Cheetah. Does this suggest that the benefit comes from the probe information rather than from CF being better?

  3. Section 6.4 provides little analysis of the experimental results, and it is not clearly explained what D1 means.

  4. Ablation studies on important parameter choices are missing, such as K=8 and H=25.

Questions for authors
  1. What do CF-state and CF-ng stand for? The paper does not explain this clearly.

  2. Does the gap between CF-WM and CF-state mainly come from accumulated error over the 25-step rollout, or from the information bottleneck of the context encoding?

1 comment
AuthorsOpenReview ↗

Response to A7xq: definitions, learned responses, and probe budget

Thank you for pointing out the unclear definitions and the missing analysis. We have clarified the terminology, reorganized the learned-model results, and added probe-budget and prediction diagnostics.

  1. CF-state, CF-ng, and access to dynamics.

CF-state generates probe responses with the true simulator and serves as a diagnostic of retrieval with accurate responses. CF-WM predicts responses from observed history. CF-ng selects the policy attached to the nearest individual memory, removing policy-group aggregation. These definitions now appear in Section 4. D1 denotes nominal dynamics; D2-D9 are the perturbations in Appendix A, also identified in Section 6.4.

The revised learned selector receives 21 observed images, 20 actions, and 20 rewards. These observations supply all query-time inputs for predicting probe responses. The selected SAC controller receives physical-state observations, as in the original setup.

  1. Learned-model performance and the source of prediction errors.

Table 4 compares methods on the original 108 queries and the same return matrix:

| Method | Walker regret | Hopper regret | | Submitted selected CF-WM | 36.96 | 11.11 | | Direct-response CF | 12.68 | 6.58 | | Source-calibrated direct-response CF | 7.98 | 2.20 |

Direct prediction uses 1,800 source response examples and a frozen visual/history representation. Walker uses WM/2, replacing the previously selected WM/0; Hopper keeps WM/1. The last two rows hold the predictor and memory bank fixed, isolating distance calibration. Hopper's calibration also adds the log-magnitude term (Appendix F.1). On new dynamics R, calibration reduces regret by 18.0% and 44.2%, respectively, although source-fixed and utility controls remain stronger (Table 5).

Appendix G compares free rollout, teacher forcing, true-image decoding, and shuffled context on six source anchors and four probes per task. At H=25, the first three conditions give state-prediction MSEs of 0.0845/0.1954/0.2078 for Walker and 0.1127/0.1322/0.0663 for Hopper. Shuffling context raises Walker MSE to 0.2478. Teacher forcing does not consistently improve prediction, and these interventions do not identify one dominant cause of the retrieval gap. Future true images are used only in the diagnostic interventions. Direct prediction avoids recursive rollout; calibration changes selection while keeping response predictions fixed.

  1. Probe budget and same-probe baselines.

Table 11 adds a six-seed comparison with KH fixed at 40:

| Task | K=2, H=20 | K=4, H=10 | K=8, H=5 | | Cheetah | 4.90 | 3.87 | 3.94 | | Walker | 26.40 | 19.67 | 20.46 | | Hopper | 9.92 | 9.71 | 8.58 | | Finger | 29.23 | 21.71 | 6.54 |

The fingerprints use states only, excluding the 25-step mean-reward channel at every horizon. Probe frequencies stay fixed. The best allocation varies by task; the controlled budget is the number of model transitions.

Your CF-ng/SysID comparison is correct. With group aggregation, CF-state has lower mean regret than SysID on Cheetah and Hopper; Walker is close, and SysID is better on Finger. We also found and corrected a missing policy-specific intercept in B4. Its corrected mean regrets are 1.12/8.26/3.12/8.22 on Cheetah/Walker/Hopper/Finger, the lowest across all four tasks (Table 1; Appendix F.2).

  1. Switching results.

Section 6.4 now explains the gains and losses, including harmful switches when dynamics stay unchanged. Appendix G reports a separate implementation check using the post-action anchor and complete branch-state restoration, including the delay queue. The repair helps two tested shifts and hurts one. Tables 6 and 9 retain the historical results; evaluating the calibrated selector during switching remains future work.

Reviewer_ygeKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes selecting policies from a fixed library using counterfactual dynamics fingerprints obtained from shared action-sequence probes. Policies are retrieved based on the similarity of predicted state and reward responses. Experiments across four control tasks show improvements over state-based retrieval when true simulator dynamics are available. Results with learned world models are substantially weaker and inconsistent. The paper also investigates selecting world models based on downstream policy-selection regret rather than prediction loss.

Critical strengths

Decision-oriented evaluation. Evaluating world models through downstream policy-selection regret rather than prediction error is a meaningful direction. The observation that prediction accuracy and decision quality can be poorly aligned is interesting.

Controlled simulator experiments. The evaluation across four tasks and six seeds provides evidence that action-response information can improve policy retrieval compared to state similarity alone.

Critical weaknesses

The proposed similarity metric lacks a clear connection to policy performance. The method compares dynamics using a small set of fixed, open-loop action sequences, while policy returns depend on the long-horizon, closed-loop state-action distribution induced by each policy. It is unclear why similarity under these probes should preserve policy-value rankings, particularly under dynamics shift. Moreover, simpler dynamics-informed approaches using the same probes, such as system identification and utility regression, are competitive or outperform the proposed method on some tasks (Table 1). This raises questions about what the proposed retrieval metric contributes beyond accessing dynamics information.

The learned-model experiments do not convincingly validate the central approach. The strongest results assume access to true environment dynamics, which is generally unavailable in the intended transfer setting. With learned models, performance deteriorates substantially: only one of three Hopper models outperforms random selection, and direct history-based methods outperform CF-WM across all evaluated Walker models (Table 3). Furthermore, selecting world models using validation regret requires policy-return labels from the held-out dynamics regimes. This considerably weakens the practical significance of the model-selection results and leaves the effectiveness of learned counterfactual retrieval under genuinely unseen dynamics insufficiently established.

Questions for authors

What justifies using similarity under fixed open-loop probes as a surrogate for similarity in closed-loop policy performance? Have the authors investigated whether policy-conditioned probes or alternative response metrics produce more reliable policy rankings?

Can the authors demonstrate consistent improvements over direct history-based retrieval when world-model selection is performed without policy-return labels from the target dynamics regimes? This seems essential to establish the practical benefit of the proposed method.

2 comments
AuthorsOpenReview ↗

Response to ygeK: probe similarity and learned retrieval

Thank you for raising the connection between probe similarity and policy returns, and the role of target-return labels in model selection. We have added a conditional analysis and new learned-model experiments to address these points.

  1. Why can open-loop probes help select a closed-loop policy?

Appendix E gives a sufficient condition for linear dynamics x'=Ax+Bu. If the shared-probe state/action contrast matrix has minimum singular value at least kappa, a state-contrast discrepancy epsilon bounds the transition-parameter discrepancy by (1+||A_m||_2) epsilon/kappa. Under a common Lipschitz reward, Lipschitz policies, and bounded visited states/actions, error propagation gives a finite-horizon return bound delta_T, including the initial-state difference. Transfer through memory m then satisfies:

Regret(q, p(m)) <= 2 delta_T + [max_p J(m,p) - J(m,p(m))].

The second term matters: even an accurate dynamics match can retrieve a controller that is poorly suited to its own source state. The derivation concerns transfer through an individual memory under the stated assumptions; extending it to nonlinear DMC and policy-group averaging requires further analysis. We also give a counterexample in which reference subtraction removes a constant drift that changes the best policy.

For the alternative-distance question, we tested positive weights on response coordinates, fitted to source-query regret (Section 4.2). The predictor, memory bank, and group retrieval rule stay fixed. Table 4 uses the original 108 queries and return matrix: calibration lowers direct-response regret from 12.68 to 7.98 on Walker and from 6.58 to 2.20 on Hopper. Hopper also gains the log-magnitude term; Appendix G separates its effect from reweighting. We have not evaluated policy-conditioned probes; the new experiments retain the shared action bank.

  1. What improves when target-return labels are excluded?

We added grid R with six new dynamics configurations per task, all excluded from training and model/configuration selection. Models and predictions were frozen before collecting R policy returns. Calibration uses 144 source-query return vectors, with the same labels supplied to the history and fingerprint utility baselines. The backbones were chosen on earlier grids; R was excluded from every selection decision (Appendix F).

On R, calibration reduces Walker regret from 15.302 to 12.540 (18.0%) and Hopper regret from 8.466 to 4.727 (44.2%). These gains occur in two of three Walker query draws and all three Hopper draws. Each task uses one frozen backbone and policy library, so the draws measure query-sampling variation.

The direct history-utility baseline remains better on R, at 10.060 for Walker and 3.755 for Hopper, matching the source-fixed and fingerprint-utility controls (Table 5). The revision therefore reports improved learned CF retrieval alongside this remaining gap. We also corrected B4's missing intercept; it now has the lowest mean regret in Table 1. The same-probe comparisons make clear that both response information and the way it is used affect selection.

Thank you for the detailed response and the additional experiments. However, my main concerns remain. The new results do not show that the proposed retrieval method is what drives performance, and the evidence in the learned-model setting is still limited. I will therefore keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and additional experiments

Thank you for the feedback on probe similarity, learned-model performance, and evaluation under new dynamics. The revision adds learned-response experiments, a conditional transfer analysis, and further diagnostics. The main changes are summarized below; our individual replies address each reviewer's questions.

  1. Improved learned retrieval on the original queries.

Table 4 uses the original 108 queries and the same policy-return matrix. Mean library regret is:

| Task | Submitted selected CF-WM | Direct-response CF | Source-calibrated direct-response CF | | Walker | 36.96 | 12.68 | 7.98 | | Hopper | 11.11 | 6.58 | 2.20 |

The direct predictor learns probe responses from 1,800 source examples using a frozen visual/history representation. Walker uses the WM/2 backbone, replacing the previously selected WM/0; Hopper uses WM/1 throughout. Comparing the direct and calibrated selectors holds the predictor and memory bank fixed and isolates distance calibration. Hopper's calibration also adds the log-magnitude term (Appendix F.1). This comparison uses the original dynamics grid, which was available during development.

  1. Confirmation on new dynamics configurations.

Grid R contains six previously unused parameter tuples per task, excluded from training and all model/configuration selection. Models and predictions were frozen before collecting R policy returns. Calibration uses 144 source-query return vectors; the history and fingerprint utility controls receive the same labels. Earlier grids informed backbone choice and method development (Appendix F).

On R, calibration reduces Walker regret from 15.302 to 12.540 (18.0%) and Hopper regret from 8.466 to 4.727 (44.2%). Walker improves in two of three query draws and Hopper in all three. Each task uses 72 queries and one frozen backbone and policy library, with five 200-step replays per query-policy pair. The draws measure query-sampling variation. Source-fixed and utility controls remain better, at 10.060 for Walker and 3.755 for Hopper (Table 5).

  1. Transfer conditions, probe diagnostics, and corrected B4.

Appendix E connects probe discrepancies to policy-return differences under linear dynamics, sufficient excitation, state coverage, and Lipschitz assumptions. The transfer bound also includes the suboptimality of the policy associated with the source memory. A counterexample shows how reference subtraction can remove information needed to distinguish policies. Table 11 and Appendix G add fixed-KH ablations and rollout, decoding, context, and switching diagnostics.

We also corrected a missing policy-specific intercept in B4. At the original regularization strength, its six-seed mean regrets are 1.12/8.26/3.12/8.22 on Cheetah/Walker/Hopper/Finger, the lowest across all four tasks. Tables 1, 8, and 10, Figure 2, and Appendix D.2 use or document the correction. The revised discussion includes these stronger baselines alongside the improvements to learned CF retrieval.

Dear Reviewers,

We're the half-way through the discussion phase. The authors have provided their response to your reviews. Please engage with the authors as soon as possible, we need to keep the conversation going!

Many thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.