- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a counterfactual dynamics-aware experience retrieval method for policy transfer. The method applies shared action sequences to query states and stored experiences, constructs reference-relative dynamics fingerprints from predicted responses, and selects a controller from a fixed policy library based on fingerprint similarity. Experiments on four DeepMind Control tasks show improvements over state-similarity-based retrieval methods. However, the theoretical justification for the proposed similarity measure is insufficient, and the experiments do not convincingly establish the reliability of the method under unseen dynamics.
Critical strengths
- The paper addresses a relevant policy transfer problem by considering action-response information rather than relying solely on state similarity.
- The method selects controllers from an existing policy library without retraining them, which is useful for policy reuse.
- The experiments include comparisons with system identification, utility regression, and state-based retrieval, as well as an evaluation using learned visual world models.
Critical weaknesses
- The central assumption lacks theoretical justification, and the connection between dynamics similarity and policy performance remains unclear. The method assumes that states or environments exhibiting similar responses to the same action sequences should favor similar control policies. However, Section 4 only defines the response fingerprints and their similarity measure, without establishing a relationship between fingerprint distance and policy performance. Two environments may respond similarly to a limited set of probe actions but behave differently under the state-action distributions induced by the deployed policies. The authors should clarify under what conditions similarity under finite action probes can support reliable policy transfer, or provide a theoretical connection between fingerprint distance and policy performance loss.
- The similarity measure in Eq. 4 has a questionable mathematical formulation. The proposed distance combines directional similarity with a logarithmic response-magnitude ratio: [\alpha\log{\left(\frac{|C_q|_2}{|C_m|_2}+\epsilon\right)}.] However, this term is not a symmetric measure of magnitude difference. Ignoring the small , it can be decomposed into the log norm of the query fingerprint minus that of the memory fingerprint. For a fixed query, the first term is constant, meaning that the formulation tends to favor memories with larger response magnitudes rather than those whose magnitudes are closer to the query. Moreover, the resulting distance can be negative and is not symmetric. The authors should justify this formulation and compare it with more appropriate alternatives, such as the absolute logarithmic magnitude difference. Otherwise, the claimed joint matching of response direction and magnitude is not sufficiently supported.
- The effectiveness of the counterfactual dynamics representation is not adequately established. Section 4.1 constructs dynamics fingerprints using eight fixed action sequences, each with a horizon of 25 steps. However, the paper does not explain why these probes are sufficient to distinguish different dynamics regimes or analyze cases where distinct dynamics parameters produce similar fingerprints. Table 2 only compares different numbers of probes, simultaneously changing the amount of response information and the computational budget. The authors should investigate the identifiability of the fingerprints and the effects of probe design, prediction horizon, and model errors on dynamics discrimination. The current experiments do not establish that the proposed fingerprints consistently capture dynamics differences relevant to policy selection.
- The experimental protocol provides limited evidence of generalization to unseen dynamics. In Section 5.2, world-model selection uses validation retrieval regret computed from policy returns in both source and held-out dynamics regimes. Although the final evaluation uses different query samples, the validation and test sets share the same dynamics parameter grid. Therefore, the results mainly demonstrate model selection within a known collection of dynamics regimes rather than generalization to genuinely unseen dynamics. The authors should evaluate the method on dynamics parameter configurations entirely excluded from model selection and avoid using target-regime policy returns during validation.
Questions for authors
The idea of using action-response information for experience retrieval is practically relevant. However, the proposed method largely relies on manually designed dynamics fingerprints and similarity matching, without sufficient theoretical justification. In particular, the magnitude term in Eq. 4 raises a clear mathematical concern, while the learned world-model results reveal substantial limitations when true dynamics information is unavailable. Furthermore, the model-selection protocol uses validation returns from target dynamics regimes, limiting the evidence for generalization to unseen environments.Overall, the current theoretical and experimental evidence is insufficient to support the central claim that the proposed method enables reliable policy transfer under unseen dynamics.
Response to VQJD: distance definition and unseen dynamics
Thank you for examining the distance definition and evaluation protocol closely. We have revised Eq. 4, added a conditional transfer analysis, and evaluated learned retrieval on a separate dynamics grid.
You are right that a signed log-ratio would favor large memory norms. Our archived local source and implementation include an absolute value. The revision makes the symmetry explicit:
alpha * |log(||C_q||_2 + epsilon) - log(||C_m||_2 + epsilon)|.
Section 4.2 defines the zero-vector convention and calls the result a dissimilarity. This numerical clarification leaves all policy choices unchanged across the 24 archived task/seed runs. The learned-selector gains come from direct response prediction and source calibration; Appendix F.1 specifies the calibrated implementation's norm convention.
Appendix E derives a sufficient condition for linear dynamics x'=Ax+Bu with sufficiently excited probes. Fingerprint discrepancy then bounds parameter discrepancy; with common Lipschitz rewards/policies and bounded visited states/actions, error propagation gives a return bound delta_T. The regret of transferring the policy associated with memory m is at most:
2 delta_T + [max_p J(m,p) - J(m,p(m))].
Prediction errors and initial-state differences enter delta_T. The second term captures how well the associated policy performs at the source memory. Applying the result requires the stated assumptions, which have not been verified for the nonlinear DMC experiments. Extending the result to group averaging also requires control of the memories in the selected group.
We give an explicit identifiability failure: for x'=x+u+b, x_0=0, and reward r(x)=x, reference-relative state and mean-reward contrasts are independent of b. Yet policies pi_+(x)=tanh(x) and pi_-(x)=-tanh(x) have three-step returns 3b+tanh(b) and 3b-tanh(b); their ranking reverses with the sign of b. Retaining the reference response resolves this example. Its usefulness more broadly is an empirical question.
For probe design, Table 11 compares (K,H)=(2,20),(4,10),(8,5) at KH=40, with state-only fingerprints and unchanged probe frequencies. The best allocation varies across tasks. Appendix G adds rollout, decoding, and context diagnostics to examine prediction error.
The original validation and test samples share a dynamics grid, as you noted. We now distinguish that comparison from grid R, which contains six new parameter tuples per task. The methods compared in Table 5 and their predictions were frozen before R policy returns were collected. Distance calibration uses 144 source-query return vectors. Earlier grids informed backbone choice and method development; R was excluded from every selection decision (Appendix F).
Table 5 reports regret reductions from 15.302 to 12.540 on Walker and from 8.466 to 4.727 on Hopper, or 18.0% and 44.2%, relative to the frozen direct-response selector. Hopper's revision includes both positive weights and a log-magnitude term; Appendix G gives the unit-weight comparison. Source-fixed and utility controls, using the same return labels, achieve lower regret at 10.060 and 3.755. Each task has one frozen backbone and library, evaluated over three query draws. The revised claims reflect both the improvement on new configurations and the remaining advantage of these baselines.