- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the issues of policy optimization, where updates are primarily based on the current policy. The authors propose Policy Laplacian Trace (PLT), a method that uses historical information to guide updates to the current policy. The paper demonstrates improvements from PLT across multiple experiments, which show the value of experience-induced geometry in policy updates.
Critical strengths
-
The paper proposes a PLT policy update framework that utilizes geometric information from historical policies to guide current policy updates. The paper also provides corresponding theoretical analysis, offering a mathematical explanation for the framework. This provides a potentially useful approach to incorporating historical information into policy optimization.
-
The experimental results seem promising, showing performance improvements on Atari, MuJoCo, StarCraft II, and LLM tasks. The paper also provides some ablation studies to analyze the role of different components. These results suggest that the proposed method may be effective across different tasks and learning settings.
Critical weaknesses
-
I have some concerns about the novelty and contribution of this paper. Both history-aware policy optimization [1] and Wasserstein-based policy optimization [2, 3] have already been studied in previous works, and this paper does not clearly demonstrate what additional advantages combining these two mechanisms provides. In its contributions, the paper emphasizes the assumption of a low-dimensional manifold, but it seems that explicitly estimating this manifold is not necessary for practical optimization. Although Appendix F.6 provides some empirical analysis, I still do not see the specific significance of this manifold assumption. The theoretical analysis mainly focuses on the proposed regularization objective and its optimization properties. I hope the authors can provide further analysis of the low-dimensional manifold assumption itself (e.g., what benefits does it bring?), as well as how the Wasserstein regularizer differs from other regularizers in terms of the information it captures. I would also like to see more specific analysis and validation of the claimed geometric properties.
-
The paper emphasizes using the geometric relationships among historical policies to guide policy updates. Compared with previous works that utilize historical information, I believe the main claim of this paper is that it not only uses multiple historical policies but also models the geometric relationships between them to guide optimization. However, the experiments do not convincingly demonstrate this point. In particular, I do not find the experiments and conclusions in Appendix F.5 convincing, as they cannot distinguish which components contribute to the performance improvements. For example, is the improvement due to the historical policies themselves, the Wasserstein geometry, or the Laplacian correction? Moreover, both EMA baselines in Appendix F.5 lack state-conditioned retrieval, making it difficult to isolate the contribution of the proposed geometric formulation.
Questions for authors
-
Could the authors clarify the specific role of the experience-induced low-dimensional manifold assumption in PLT? Since the practical algorithm does not explicitly estimate this manifold, how does this assumption contribute to the optimization process? What evidence demonstrates that the geometric structure provides additional benefits beyond conventional history-aware regularization?
-
Could the authors further clarify how the performance improvements can be attributed to the proposed geometric formulation? In particular, the EMA baselines in Appendix F.5 do not use state-conditioned retrieval. Would it be possible to compare different regularization methods under the same historical memory and retrieval settings to better isolate the contribution of Wasserstein geometry?
-
In Proposition H.13, the squared Wasserstein distance is claimed to be geodesically strongly convex on the general Wasserstein space. However, this property does not hold in general for dimensions greater than one. Could the authors clarify the assumptions under which the claimed contraction guarantee holds?
11 comments
Response for Weakness 2 & Q1
W2. Which Components Contribute to Performance?
We added controls separating reference construction, update rule, retrieval, and transport aggregation. All comparisons use DonkeyKong-v5, 3 training seeds, and 30M steps.
Reference construction and update rule. The history-based variants share the memory-construction and retrieval protocols:
| Configuration | Peak return ± episode SD ↑ |
|---|---|
| PPO + KL toward the rollout policy | 4560 ± 48.99 |
| PPO + KL toward the retrieved historical mixture | 4100 ± 228.04 |
| PPO + KL toward the semantic transport-aligned reference | 4300 ± 141.42 |
| PLT: semantic transport alignment + anchored clipping | 4220 ± 74.83 |
SD is across five evaluation episodes at the peak checkpoint. Replacing the rollout reference with retrieved history lowers peak return by 460 points. Transport alignment recovers 200 points while retaining PPO-Clip and the KL penalty weight. Replacing that KL regularization with anchored clipping then lowers the peak by 80 points. Thus, transport-based reference construction helps within these history-based controls, but anchored clipping provides no observed gain; neither exceeds the rollout-policy KL baseline.
Retrieval and neighbor-wise transport. As detailed in W1.1, age-matched random retrieval reduces late return from 3603.33 to 190.00, despite retaining neighbor weights and the transport update. With state-conditioned retrieval retained, replacing mixture transport with weighted per-neighbor costs increases late return to 4636.67 and normalized AUC from 2802.37 to 3737.63. This directly supplies the previously missing mixture-versus-neighbor-wise comparison.
Ground cost. The comparison in W1.3 holds anchored clipping fixed: replacing the flat 0/1 cost with the semantic cost increases peak return from 4060 to 4220.
Together, these controls support state-conditioned retrieval, transport-based reference construction, and neighbor-wise aggregation in the tested setting, with a smaller observed benefit from the semantic cost. They do not support an additional gain from anchored clipping. These single-seed performance comparisons neither establish significance across training seeds nor directly measure retention.
Q1. What Role Does the Low-Dimensional Manifold Assumption Play in PLT?
As discussed in W1.2, the assumption motivates local coverage by finite history: when policy variation is locally low dimensional, sufficiently representative historical policies may guide updates without reconstructing global coordinates. This requires adequate coverage and retrieval neighborhoods consistent with behavioral similarity; low dimensionality alone is insufficient.
The assumption is not required to define the finite-graph objective, and our theory provides no intrinsic-dimension-dependent performance bound. Appendix F.6 will therefore be presented as exploratory evidence, with the measured objects, estimators, snapshot counts, and reference dimensions specified.
A further distinction is that Hypothesis 1 concerns policy trajectories at a fixed input, whereas the implementation retrieves history through keys associated with similar inputs. The former does not establish the validity of the latter. We will make this gap explicit and assess agreement between state-key neighborhoods and policy-Wasserstein neighborhoods.
Response for Q2 & Q3
Q2. How Can Performance Gains Be Attributed to the Geometric Formulation?
We added controls that retain state-conditioned retrieval while changing the transport construction, separating geometric choices from access to historical information.
As reported in W2, both reference variants use the same memory and retrieval protocols, PPO ratio, and KL penalty weight. Replacing the retrieved-mixture reference with a semantic transport-aligned reference increases peak return from 4100 to 4300. Unlike the EMA comparison, this control retains state conditioning in both arms; it tests reference construction, not a direct substitution of Wasserstein for KL regularization.
The comparison in W1.1 retains retrieval, neighbor weighting, semantic action cost, and anchored clipping, replacing transport to the historical mixture with weighted per-neighbor transport costs. Late return increases from 3603.33 to 4636.67 (+28.68%), and normalized AUC from 2802.37 to 3737.63 (+33.37%). This supplies direct evidence for neighbor-wise aggregation beyond mixture-based transport. The separate age-matched random-retrieval control sharply reduces performance, identifying neighbor selection as another contributing component.
As detailed in W1.3, replacing a flat 0/1 cost with the semantic action cost under anchored clipping increases peak return from 4060 to 4220. This tests the ground cost, separately from how transport costs are aggregated.
Together, these controls support specific geometric design choices on DonkeyKong-v5. They do not establish general gains across seeds or tasks, validate the manifold assumption, or imply contraction of the implemented training process. The update-rule comparison in W2 also shows no additional gain from anchored clipping.
Q3. Under What Assumptions Does the Contraction Guarantee in Proposition H.13 Hold?
The original geodesic strong-convexity argument does not establish contraction in general multidimensional Wasserstein spaces. We will withdraw that general claim and replace it with the following restricted result.
Let the policy family , equipped with , be isometric to a closed convex subset of a Hilbert space. Fix historical references and weights with . Define
For , solve the following problem exactly:
\left[ \Phi(q)+\frac{W_2^2(q,p)}{2\tau} \right].$$ With the same references and weights for both inputs, $$W_2(T_\tau(p),T_\tau(p'))=\frac{1}{1+\tau}W_2(p,p').$$ Let $\Psi$ be the isometry, $x=\Psi(p)$, and $b=\sum_i w_i\Psi(q_i)$. The weighted variance identity gives $$\Phi(q)=\frac{1}{2}\|\Psi(q)-b\|^2+K,$$ where $K$ is independent of $q$. The unique proximal minimizer in these coordinates is $$\Psi(T_\tau(p))=\frac{x+\tau b}{1+\tau}.$$ Convexity ensures feasibility; subtracting the solutions for two inputs gives the stated contraction factor. This setting includes one-dimensional distributions with finite second moments under the quantile representation, and diagonal Gaussian distributions in a common coordinate system, including zero variances. **Low intrinsic dimension alone does not imply the required condition.** The result does not automatically extend to fixed finite action supports, changing references or weights, KL anchoring, entropic transport, or approximate solvers. We will revise **H.14** accordingly: with fixed history and a fixed positive $\tau$, repeated application of this exact mapping converges to the historical Wasserstein barycenter within $C$. Changing history or approximate solutions requires additional perturbation analysis. This is a guarantee for the specified **Wasserstein-proximal module**, not for the implemented KL-anchored fusion or the full PLT training process.
Response for Weaknesses 1
Thank you for asking us to clarify PLT’s novelty and manifold assumption, isolate the contribution of geometry, and correct the contraction claim.
W1.1. Combining Historical References with Transport Geometry
PLT uses state-conditioned historical policies as behavioral references, with transport costs guiding updates relative to them. We added neighbor-selection and transport-aggregation controls on DonkeyKong-v5 (30M training steps per run).
Random retrieval matches the selected neighbors’ insertion iterations and retains their weights. Neighbor-wise transport replaces transport to the weighted mixture with weighted per-neighbor costs. Both retain the semantic action cost and anchored-clipping update. Late return averages evaluations at 27–29.5M steps. Normalized AUC is the trapezoidal return integral over 0–29.5M divided by 29.5M.
Random retrieval reduces late return by 94.73%. With state-conditioned retrieval retained, neighbor-wise transport improves late return by 28.68% and AUC by 33.37%. These controls support contributions from relevant neighbor selection and individual neighbor transport costs within this task. General advantages over prior methods, retention/relearning benefits, and significance across training seeds remain unestablished.
W1.2. Role of the Low-Dimensional Manifold Assumption
The assumption motivates coverage of relevant policy variation by finite memory. If historical policies concentrate near a locally low-dimensional set under a behavioral distance, representative samples may cover that variation without reconstructing global coordinates. This requires adequate coverage and retrieval neighborhoods consistent with behavioral similarity. Low dimensionality alone is insufficient.
The finite-graph regularizer is well-defined without this assumption. Our proximal analysis requires separate conditions, and our theory provides no dimension-dependent performance bound.
We will present Appendix F.6 as exploratory structural evidence, specifying the measured objects, estimators, snapshot counts, and policy-output dimensions. We will assess agreement between state-key retrieval and policy-Wasserstein neighborhoods and clarify the geometric assumptions for continuous and discrete policies.
W1.3. Information Captured by Wasserstein Regularization
Wasserstein regularization encodes cross-action relationships through the ground cost. KL compares probabilities at corresponding actions. Transport accounts for how far probability mass moves, so equal KL divergence from a historical reference can correspond to different transport costs. The value of this information depends on a behaviorally meaningful ground cost.
With history, retrieval, and anchored clipping fixed, replacing a flat 0/1 cost with the semantic cost increased DonkeyKong-v5 peak return from 4060 ± 300.67 to 4220 ± 74.83. SDs describe five evaluation episodes at the peak checkpoint. This provides limited evidence for the semantic cost in this setting.
We will specify each task’s action representation, cost matrix, and normalization, and add an equal-KL, different-transport example. We will also clarify that the LM embedding-mean proxy captures only first-moment displacement and cannot distinguish distributions with identical embedding means.
W1.4. Scope and Validation of the Geometric Claims
We will distinguish the idealized local energy, exact proximal map, and implemented approximation. The strong-convexity argument fails in general multidimensional Wasserstein spaces. We will replace the general contraction claim in Appendix H.13 with the restricted result and explicit assumptions in Q3.
The controls in W1.1 and W1.3 assess transport aggregation and ground-cost choices. Our frozen-history LM diagnostic compares the optimized embedding-mean proxy and implemented gate under the same transport-proximal objective. Their mean objective gaps from a numerical reference are 0.001184 and 0.617993, respectively, with the gate having a larger gap at 71 of 72 positions. These measurements use truncated, renormalized supports, leaving approximation accuracy over the full vocabulary unassessed.
We will restrict exact-proximal guarantees to the proved setting. Extensions to entropic transport, historical mixing, projection, and the outer PPO update require separate analysis.