- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a positive semidefinite matrix representation for trajectory segments, which aggregates local state changes, actions, rewards, and related information into matrix descriptors, while using matrix addition to compose trajectory segments. Based on this representation, the authors propose MSRL, which pretrains a matrix-to-value branch on source tasks and combines segment retrieval with obstruction filtering for policy learning on target tasks.
Critical strengths
-
The matrix-based representation of trajectory segments has some novelty, and the overall method is relatively unified in its design.
-
The experiments include negative controls such as random initialization, shuffled source value, and unaligned lift, which help assess whether the observed gains actually come from meaningful pretraining.
Critical weaknesses
-
The composition property essentially follows from the additivity of while completeness is established only for the specifically defined affine-quadratic function class The obstruction theorem is also largely equivalent to stating that a point does not belong to the reachable set if its distance to that set is greater than zero. These results are formally complete, but they do not provide sufficiently strong guarantees on RL performance or generalization.
-
The experiments explicitly use a simplified and remove the Kronecker action-position blocks from the theoretical definition. In addition, the segment length in the target-control library is actually fixed to 1 transition. Therefore, the multi-step segment composition, gluing, and related theory emphasized in the main text are not directly validated in the main control experiments.
-
The paper states that it reports final performance, success rate, and steps-to-threshold, but I did not find the corresponding complete results.
-
The main source-transfer comparisons include Traj2Value-RNN-FT, Traj2Value-Transformer-FT, and Subgraph-GNN-to-Value-FT, which are adapted variants constructed by the authors rather than standard algorithms from prior work.
-
There are multiple clear editing and formatting issues in both the references and the main text. For example, the HER authors are listed as “OpenAI Pieter Abbeel,” which is clearly incorrect. Conference and journal names are also inconsistent, such as “International Conference on Machine Learning / International conference on machine learning,” “PMLR / Pmlr,” and “Nature / nature.” The Ravindran (2003) entry also lacks complete publication information. Appendix E.1 further contains nearly duplicated environment-description paragraphs.
Questions for authors
See Weaknesses.
2 comments
Response to numerical reporting and the baseline design.
We agree that complete final-return, success-rate, and steps-to-threshold tables are not included. The primary reported metric is normalized few-shot AUC (Table 2, five target families and five seeds), complemented by full-budget learning curves for AntMaze, Walker2d, and Hopper (Figure 2). Source-initialized MSRL obtains mean AUC, compared with for the matched from-scratch model and for TD-MPC-PT+FT. We have revised the manuscript to restrict numerical claims to these reported results.
Because success rates and steps-to-threshold require task-specific criteria or thresholds, we do not infer them from the reported AUC values or mean learning curves.
We make no separate superiority claim for these unreported metrics.
We clarify the distinction between published RL algorithms and controlled representation adaptations used in our experiments. PPO, SAC, TD3, MBPO, DreamerV3, and TD-MPC are published RL algorithms. Traj2Value-RNN-FT, Traj2Value-Transformer-FT, and Subgraph-GNN-to-Value-FT are intentionally constructed representation controls, not claimed to be standalone published algorithms. Their role is to hold the source-to-target protocol fixed while varying the representation; their validity as controls does not depend on each variant having been separately published. We clarify the groups and initialized components in the table captions and Appendix E.
Manuscript revision. We align every metric statement with the displayed numerical results and explicitly label controlled representation baselines as such.
Response to citation and editorial accuracy.
We thank the reviewer for identifying these bibliographic and editorial inconsistencies. We have corrected the HER author field, the Ravindran--Barto IJCAI 2003 entry, and inconsistent venue capitalization, and have streamlined the explanatory wording in Appendix E.1.
Response to the mathematical contribution and its control implications.
We would like to clarify the distinct roles of the structural results and the empirical control evaluation. The former characterize the descriptor under explicit assumptions rather than establishing unconditional RL improvement. The contribution is an explicit representation in which useful operations on observed transitions are exact and compatible: prefix accumulation, valid concatenation, and nonduplicating overlap assembly occur in one fixed-dimensional PSD space. Completeness and minimal sufficiency establish a precise information boundary: equality of descriptors is equivalent to equality of every additive affine-quadratic functional induced by the chosen lift, and any other admissible descriptor sufficient for that class must determine M. This is functional sufficiency, not minimum storage dimension or preservation of arbitrary history.
The value-theoretic results play a different role. Given matrix sufficiency, lifted Bellman compatibility, and differentiability, the matrix-conditioned value admits a local first-order expansion. We do not claim global linearity for finite segments, a guarantee that arbitrary compositions improve return, or universally superior transfer. In fully observed Markov tasks, the optimal value may be independent of history; accordingly, practical finite-sample benefits must come from the learned inductive bias and are an empirical question. Likewise, the exact reachable-set obstruction is a mathematical criterion; its learned implementation gω is a heuristic filter, not a certified oracle. The theory states what the representation guarantees and what must additionally hold for the value interface. The experiments separately measure learning performance.
Manuscript revision. Sections 3--4 distinguish exact descriptor properties, conditional value statements, and empirical control outcomes, without promoting a representation identity into an unconditional RL theorem.
Response to multi-step composition and the one-step library.
A key distinction is between the one-step retrieval library and the multi-step history summarized by the critic. The reported library contains one-transition entries, whereas the critic state is Zt=∑s=0t−1ψsψs⊤ and therefore aggregates the entire executed prefix. In particular, Zt+1=Zt+ψtψt⊤ performs exact repeated composition at every environment step. Thus the evaluated method does use multi-step historical statistics, even though candidate retrieval/lookahead is limited to one transition. A one-step library is a computationally bounded design choice; it is not a test of the performance gains from retrieving arbitrary long segments or performing learned overlap gluing, and we do not claim it is.
Independent source-side checks do provide descriptor-level evidence: GraphMotif directed-motif/reversal retrieval is 0.939, MicroGrid obstruction AUROC is 0.889, and additive-consistency errors are small (Appendix E). These support the structural interpretation, not an unreported multi-step planning experiment. The appropriate conclusion is that the current targets validate prefix accumulation, one-step retrieval, and source initialization, while the more general multi-segment operations are established algebraically.
Manuscript revision. Sections 5--6 and Appendix E now distinguish executed-prefix horizon, retrieval horizon, and general representation-level gluing.