acceptodds
Under review as a conference paper at ICLR 2027

Laplacian Flows for Policy Learning

Abstract

Modern AI systems improve task performance through local policy updates, yet gains on the current training objective can come at the expense of previously acquired behaviours, a risk that conservative single-step updates alone do not eliminate. To guide adaptation with a reference to accumulated experience, we draw on geometric measure theory and model historical policies as samples from an experience-induced, locally low-dimensional manifold in Wasserstein space. We introduce Policy Laplacian Trace (PLT), an evolving trajectory map that uses geometric relations among relevant historical policies to regularise subsequent updates. Unlike Wasserstein constraints imposed only between successive iterates, PLT relates each candidate update to multiple historical policies. A history-induced graph Dirichlet energy translates these geometric relations into a Laplacian-type correction to the task-driven update, bringing trajectory-level geometric context into local optimisation. Learning shapes the map, while its evolving graph geometry guides subsequent policy updates. We instantiate PLT in reinforcement learning and language-model training, using a scalable semantic-transport proxy for the latter. Experiments with PPO and MAPPO on Atari, MuJoCo, and StarCraft II show improvements in final performance and sample efficiency. Controlled language-model stress tests show gains in robustness to contradictory evidence, long-range factual recall, and few-shot generalisation. On the recall task, Qwen3-4B improves weighted token accuracy by 4.83%, with a 3.56% increase in median training-step time under the tested configuration. This feedback between policy evolution and its geometric map suggests a potential role for experience-induced geometry in recursive learning.

Primary area:
learning theory
Keywords:
Policy Learning, Geometry Measure Theory, Topology

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RFGKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the issues of policy optimization, where updates are primarily based on the current policy. The authors propose Policy Laplacian Trace (PLT), a method that uses historical information to guide updates to the current policy. The paper demonstrates improvements from PLT across multiple experiments, which show the value of experience-induced geometry in policy updates.

Critical strengths
  1. The paper proposes a PLT policy update framework that utilizes geometric information from historical policies to guide current policy updates. The paper also provides corresponding theoretical analysis, offering a mathematical explanation for the framework. This provides a potentially useful approach to incorporating historical information into policy optimization.

  2. The experimental results seem promising, showing performance improvements on Atari, MuJoCo, StarCraft II, and LLM tasks. The paper also provides some ablation studies to analyze the role of different components. These results suggest that the proposed method may be effective across different tasks and learning settings.

Critical weaknesses
  1. I have some concerns about the novelty and contribution of this paper. Both history-aware policy optimization [1] and Wasserstein-based policy optimization [2, 3] have already been studied in previous works, and this paper does not clearly demonstrate what additional advantages combining these two mechanisms provides. In its contributions, the paper emphasizes the assumption of a low-dimensional manifold, but it seems that explicitly estimating this manifold is not necessary for practical optimization. Although Appendix F.6 provides some empirical analysis, I still do not see the specific significance of this manifold assumption. The theoretical analysis mainly focuses on the proposed regularization objective and its optimization properties. I hope the authors can provide further analysis of the low-dimensional manifold assumption itself (e.g., what benefits does it bring?), as well as how the Wasserstein regularizer differs from other regularizers in terms of the information it captures. I would also like to see more specific analysis and validation of the claimed geometric properties.

  2. The paper emphasizes using the geometric relationships among historical policies to guide policy updates. Compared with previous works that utilize historical information, I believe the main claim of this paper is that it not only uses multiple historical policies but also models the geometric relationships between them to guide optimization. However, the experiments do not convincingly demonstrate this point. In particular, I do not find the experiments and conclusions in Appendix F.5 convincing, as they cannot distinguish which components contribute to the performance improvements. For example, is the improvement due to the historical policies themselves, the Wasserstein geometry, or the Laplacian correction? Moreover, both EMA baselines in Appendix F.5 lack state-conditioned retrieval, making it difficult to isolate the contribution of the proposed geometric formulation.

Questions for authors
  1. Could the authors clarify the specific role of the experience-induced low-dimensional manifold assumption in PLT? Since the practical algorithm does not explicitly estimate this manifold, how does this assumption contribute to the optimization process? What evidence demonstrates that the geometric structure provides additional benefits beyond conventional history-aware regularization?

  2. Could the authors further clarify how the performance improvements can be attributed to the proposed geometric formulation? In particular, the EMA baselines in Appendix F.5 do not use state-conditioned retrieval. Would it be possible to compare different regularization methods under the same historical memory and retrieval settings to better isolate the contribution of Wasserstein geometry?

  3. In Proposition H.13, the squared Wasserstein distance is claimed to be geodesically strongly convex on the general Wasserstein space. However, this property does not hold in general for dimensions greater than one. Could the authors clarify the assumptions under which the claimed contraction guarantee holds?

11 comments
AuthorsOpenReview ↗

Response for Weaknesses 1

Thank you for asking us to clarify PLT’s novelty and manifold assumption, isolate the contribution of geometry, and correct the contraction claim.

W1.1. Combining Historical References with Transport Geometry

PLT uses state-conditioned historical policies as behavioral references, with transport costs guiding updates relative to them. We added neighbor-selection and transport-aggregation controls on DonkeyKong-v5 (30M training steps per run).

ConfigurationLate return ↑Normalized AUC ↑
State-conditioned retrieval + mixture transport3603.332802.37
Age-matched random retrieval + mixture transport190.00182.37
State-conditioned retrieval + neighbor-wise transport4636.673737.63

Random retrieval matches the selected neighbors’ insertion iterations and retains their weights. Neighbor-wise transport replaces transport to the weighted mixture with weighted per-neighbor costs. Both retain the semantic action cost and anchored-clipping update. Late return averages evaluations at 27–29.5M steps. Normalized AUC is the trapezoidal return integral over 0–29.5M divided by 29.5M.

Random retrieval reduces late return by 94.73%. With state-conditioned retrieval retained, neighbor-wise transport improves late return by 28.68% and AUC by 33.37%. These controls support contributions from relevant neighbor selection and individual neighbor transport costs within this task. General advantages over prior methods, retention/relearning benefits, and significance across training seeds remain unestablished.

W1.2. Role of the Low-Dimensional Manifold Assumption

The assumption motivates coverage of relevant policy variation by finite memory. If historical policies concentrate near a locally low-dimensional set under a behavioral distance, representative samples may cover that variation without reconstructing global coordinates. This requires adequate coverage and retrieval neighborhoods consistent with behavioral similarity. Low dimensionality alone is insufficient.

The finite-graph regularizer is well-defined without this assumption. Our proximal analysis requires separate conditions, and our theory provides no dimension-dependent performance bound.

We will present Appendix F.6 as exploratory structural evidence, specifying the measured objects, estimators, snapshot counts, and policy-output dimensions. We will assess agreement between state-key retrieval and policy-Wasserstein neighborhoods and clarify the geometric assumptions for continuous and discrete policies.

W1.3. Information Captured by Wasserstein Regularization

Wasserstein regularization encodes cross-action relationships through the ground cost. KL compares probabilities at corresponding actions. Transport accounts for how far probability mass moves, so equal KL divergence from a historical reference can correspond to different transport costs. The value of this information depends on a behaviorally meaningful ground cost.

With history, retrieval, and anchored clipping fixed, replacing a flat 0/1 cost with the semantic cost increased DonkeyKong-v5 peak return from 4060 ± 300.67 to 4220 ± 74.83. SDs describe five evaluation episodes at the peak checkpoint. This provides limited evidence for the semantic cost in this setting.

We will specify each task’s action representation, cost matrix, and normalization, and add an equal-KL, different-transport example. We will also clarify that the LM embedding-mean proxy captures only first-moment displacement and cannot distinguish distributions with identical embedding means.

W1.4. Scope and Validation of the Geometric Claims

We will distinguish the idealized local energy, exact proximal map, and implemented approximation. The strong-convexity argument fails in general multidimensional Wasserstein spaces. We will replace the general contraction claim in Appendix H.13 with the restricted result and explicit assumptions in Q3.

The controls in W1.1 and W1.3 assess transport aggregation and ground-cost choices. Our frozen-history LM diagnostic compares the optimized embedding-mean proxy and implemented gate under the same transport-proximal objective. Their mean objective gaps from a numerical reference are 0.001184 and 0.617993, respectively, with the gate having a larger gap at 71 of 72 positions. These measurements use truncated, renormalized supports, leaving approximation accuracy over the full vocabulary unassessed.

We will restrict exact-proximal guarantees to the proved setting. Extensions to entropic transport, historical mixing, projection, and the outer PPO update require separate analysis.

AuthorsOpenReview ↗

Response for Weakness 2 & Q1

W2. Which Components Contribute to Performance?

We added controls separating reference construction, update rule, retrieval, and transport aggregation. All comparisons use DonkeyKong-v5, 3 training seeds, and 30M steps.

Reference construction and update rule. The history-based variants share the memory-construction and retrieval protocols:

ConfigurationPeak return ± episode SD ↑
PPO + KL toward the rollout policy4560 ± 48.99
PPO + KL toward the retrieved historical mixture4100 ± 228.04
PPO + KL toward the semantic transport-aligned reference4300 ± 141.42
PLT: semantic transport alignment + anchored clipping4220 ± 74.83

SD is across five evaluation episodes at the peak checkpoint. Replacing the rollout reference with retrieved history lowers peak return by 460 points. Transport alignment recovers 200 points while retaining PPO-Clip and the KL penalty weight. Replacing that KL regularization with anchored clipping then lowers the peak by 80 points. Thus, transport-based reference construction helps within these history-based controls, but anchored clipping provides no observed gain; neither exceeds the rollout-policy KL baseline.

Retrieval and neighbor-wise transport. As detailed in W1.1, age-matched random retrieval reduces late return from 3603.33 to 190.00, despite retaining neighbor weights and the transport update. With state-conditioned retrieval retained, replacing mixture transport with weighted per-neighbor costs increases late return to 4636.67 and normalized AUC from 2802.37 to 3737.63. This directly supplies the previously missing mixture-versus-neighbor-wise comparison.

Ground cost. The comparison in W1.3 holds anchored clipping fixed: replacing the flat 0/1 cost with the semantic cost increases peak return from 4060 to 4220.

Together, these controls support state-conditioned retrieval, transport-based reference construction, and neighbor-wise aggregation in the tested setting, with a smaller observed benefit from the semantic cost. They do not support an additional gain from anchored clipping. These single-seed performance comparisons neither establish significance across training seeds nor directly measure retention.

Q1. What Role Does the Low-Dimensional Manifold Assumption Play in PLT?

As discussed in W1.2, the assumption motivates local coverage by finite history: when policy variation is locally low dimensional, sufficiently representative historical policies may guide updates without reconstructing global coordinates. This requires adequate coverage and retrieval neighborhoods consistent with behavioral similarity; low dimensionality alone is insufficient.

The assumption is not required to define the finite-graph objective, and our theory provides no intrinsic-dimension-dependent performance bound. Appendix F.6 will therefore be presented as exploratory evidence, with the measured objects, estimators, snapshot counts, and reference dimensions specified.

A further distinction is that Hypothesis 1 concerns policy trajectories at a fixed input, whereas the implementation retrieves history through keys associated with similar inputs. The former does not establish the validity of the latter. We will make this gap explicit and assess agreement between state-key neighborhoods and policy-Wasserstein neighborhoods.

AuthorsOpenReview ↗

Response for Q2 & Q3

Q2. How Can Performance Gains Be Attributed to the Geometric Formulation?

We added controls that retain state-conditioned retrieval while changing the transport construction, separating geometric choices from access to historical information.

As reported in W2, both reference variants use the same memory and retrieval protocols, PPO ratio, and KL penalty weight. Replacing the retrieved-mixture reference with a semantic transport-aligned reference increases peak return from 4100 to 4300. Unlike the EMA comparison, this control retains state conditioning in both arms; it tests reference construction, not a direct substitution of Wasserstein for KL regularization.

The comparison in W1.1 retains retrieval, neighbor weighting, semantic action cost, and anchored clipping, replacing transport to the historical mixture with weighted per-neighbor transport costs. Late return increases from 3603.33 to 4636.67 (+28.68%), and normalized AUC from 2802.37 to 3737.63 (+33.37%). This supplies direct evidence for neighbor-wise aggregation beyond mixture-based transport. The separate age-matched random-retrieval control sharply reduces performance, identifying neighbor selection as another contributing component.

As detailed in W1.3, replacing a flat 0/1 cost with the semantic action cost under anchored clipping increases peak return from 4060 to 4220. This tests the ground cost, separately from how transport costs are aggregated.

Together, these controls support specific geometric design choices on DonkeyKong-v5. They do not establish general gains across seeds or tasks, validate the manifold assumption, or imply contraction of the implemented training process. The update-rule comparison in W2 also shows no additional gain from anchored clipping.

Q3. Under What Assumptions Does the Contraction Guarantee in Proposition H.13 Hold?

The original geodesic strong-convexity argument does not establish contraction in general multidimensional Wasserstein spaces. We will withdraw that general claim and replace it with the following restricted result.

Let the policy family CC, equipped with W2W_2, be isometric to a closed convex subset of a Hilbert space. Fix historical references qi∈Cq_i \in C and weights wi≥0w_i \geq 0 with ∑iwi=1\sum_i w_i=1. Define

Φ(p)=12∑iwiW22(p,qi).\Phi(p)=\frac{1}{2}\sum_i w_i W_2^2(p,q_i).

For τ>0\tau>0, solve the following problem exactly:

\left[ \Phi(q)+\frac{W_2^2(q,p)}{2\tau} \right].$$ With the same references and weights for both inputs, $$W_2(T_\tau(p),T_\tau(p'))=\frac{1}{1+\tau}W_2(p,p').$$ Let $\Psi$ be the isometry, $x=\Psi(p)$, and $b=\sum_i w_i\Psi(q_i)$. The weighted variance identity gives $$\Phi(q)=\frac{1}{2}\|\Psi(q)-b\|^2+K,$$ where $K$ is independent of $q$. The unique proximal minimizer in these coordinates is $$\Psi(T_\tau(p))=\frac{x+\tau b}{1+\tau}.$$ Convexity ensures feasibility; subtracting the solutions for two inputs gives the stated contraction factor. This setting includes one-dimensional distributions with finite second moments under the quantile representation, and diagonal Gaussian distributions in a common coordinate system, including zero variances. **Low intrinsic dimension alone does not imply the required condition.** The result does not automatically extend to fixed finite action supports, changing references or weights, KL anchoring, entropic transport, or approximate solvers. We will revise **H.14** accordingly: with fixed history and a fixed positive $\tau$, repeated application of this exact mapping converges to the historical Wasserstein barycenter within $C$. Changing history or approximate solutions requires additional perturbation analysis. This is a guarantee for the specified **Wasserstein-proximal module**, not for the implemented KL-anchored fusion or the full PLT training process.
Reviewer_RC9SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Policy Laplacian Trace (PLT), a method that uses historical policy distributions to guide subsequent learning updates. The motivation is that restricting individual updates does not necessarily prevent cumulative behavioral drift. PLT maintains a memory of past policies, retrieves entries associated with similar states or contexts, and uses Wasserstein-based regularization together with KL anchoring to encourage consistency with relevant historical behavior. The method is applied to PPO and MAPPO in reinforcement learning and to supervised language-model training through a scalable semantic-distance approximation and gated memory fusion. Experiments cover Atari, MuJoCo, and StarCraft II, alongside controlled language-model tasks involving contradictory evidence, factual recall, and few-shot adaptation. The authors report improvements in task performance and sample efficiency, including a 4.83-percentage-point increase in weighted token accuracy for Qwen3-4B on the recall task.

Critical strengths

A compelling use of policy history to guide learning. PLT uses multiple relevant historical policies as references for subsequent updates, extending beyond constraints that only compare consecutive policies. This provides a useful perspective on controlling cumulative behavioral drift and connects historical-policy information to a concrete graph-based regularization framework.

Broad empirical evaluation with encouraging practical results. The paper evaluates PLT across Atari, MuJoCo, and StarCraft II, alongside controlled language-model tasks involving contradictory evidence, long-range recall, and few-shot adaptation. The reported gains across these settings support its potential applicability, while runtime measurements help assess practicality: on the Qwen3-4B recall task, weighted token accuracy improves by 4.83 percentage points with a 3.56% increase in median training-step time.

Critical weaknesses

Put in questions below

Questions for authors

I apologize in advance if I have misread or misinterpreted anything, but I have two questions:

  1. Do the experiments isolate the benefit of Wasserstein–Laplacian geometry from that of state-conditioned historical retrieval?
  2. What theoretical or empirical evidence supports treating the LLM implementation’s semantic-distance proxy and gated interpolation as approximations to the proposed Wasserstein–Laplacian formulation?
3 comments
AuthorsOpenReview ↗

Response for all Questions

Thank you for asking us to isolate the contribution of Wasserstein–Laplacian geometry and justify our LLM approximations.

Q1. Separating Wasserstein–Laplacian Geometry from Historical Retrieval

We added controls for neighbor selection, transport reference construction, and neighbor-wise aggregation on DonkeyKong-v5 (3 training seeds, 30M training steps).

Retrieval and transport aggregation. Reference PLT uses state-conditioned retrieval, transport to the weighted historical mixture, and anchored clipping. Random retrieval substitutes random entries matched by insertion iteration, retaining the original weights. Neighbor-wise transport retains state-conditioned retrieval and uses a weighted sum of individual neighbor transport costs. All configurations retain the semantic action cost and anchored-clipping update.

ConfigurationLate return ↑Normalized AUC ↑
State-conditioned retrieval + mixture transport3603.332802.37
Age-matched random retrieval + mixture transport190.00182.37
State-conditioned retrieval + neighbor-wise transport4636.673737.63

Late return averages six evaluations at steps 27–29.5M. Normalized AUC is the trapezoidal return integral over 0–29.5M divided by 29.5M. Random retrieval reduces late return by 94.73%. With state-conditioned retrieval retained, neighbor-wise transport improves late return by 28.68% and normalized AUC by 33.37%. These controls separately test neighbor selection and transport aggregation.

Transport reference and action geometry. With retrieval, the PPO-Clip update, and KL penalty weight fixed, replacing the retrieved-mixture reference with a semantic transport-aligned reference increases peak return from 4100 ± 228.04 to 4300 ± 141.42. This tests reference construction under the same KL penalty. Separately, replacing a flat 0/1 action cost with the semantic cost under anchored clipping raises peak return from 4060 ± 300.67 to 4220 ± 74.83. SDs are over five evaluation episodes at the peak checkpoint.

These results support a contribution from transport beyond retrieval in this setting. Statistical significance across training seeds and exact Wasserstein–Laplacian proximal guarantees remain unestablished.

Q2. Justifying the LLM Semantic-Distance Proxy and Gated Interpolation

The mathematical connection is local graph smoothing in first-moment space. We added diagnostics for the distance proxy and gated update.

What the implementation preserves. For fixed token embeddings eae_a, define m(p)=∑ap(a)eam(p)=\sum_a p(a)e_a. Under the ground cost c(a,b)=∥ea−eb∥2c(a,b)=\|e_a-e_b\|^2, Jensen's inequality gives

∥m(p)−m(q)∥2≤W2,E2(p,q).\|m(p)-m(q)\|^2 \le W_{2,E}^2(p,q).

The proxy captures mean displacement but can miss distributional differences beyond the mean. For fixed retrieved policies qiq_i and nonnegative weights summing to one, let mˉ=∑iwim(qi)\bar m=\sum_i w_i m(q_i). The projected graph energy is

=\frac{1}{2}\|m-\bar m\|^2+C,$$ where $C$ is independent of $m$. The gated update in Eq. (35), $$p^+=(1-g)p+g\sum_i w_iq_i,\qquad 0\le g\le1,$$ satisfies $m(p^+)-\bar m=(1-g)(m(p)-\bar m)$, multiplying the excess projected energy by $(1-g)^2$. This establishes smoothing for a fixed local graph. Wasserstein contraction of the full distributions and equivalence to Eq. (34) remain unproved. **Distance-proxy diagnostic.** Across **72 WikiText-2 validation positions** with memory hits, within-seed rank correlations between the mean proxy and exact transport were **0.899–0.959**, with **83 of 828** position pairs reversing order. The proxy thus preserves rankings imperfectly. Calculations used truncated, renormalized distributions retaining only **17.82%** of historical probability mass on average. **Update diagnostic.** At the same positions, we fixed the base distribution and retrieved history and compared the optimized mean-proxy solution with the actual gated output on shared, renormalized supports of **7–12 tokens**. Both were scored under the same transport-proximal objective against a numerical reference. | Candidate | Mean transport-objective gap ↓ | | --- | ---: | | Optimized embedding-mean proxy | **0.001184** | | Actual gated output | 0.617993 | The gate had a larger gap at **71 of 72 positions**, indicating less accurate objective values than the optimized proxy. These gaps concern truncated supports and the training gate. Distributional proximity over the full vocabulary remains unassessed. We will describe the implementation as **historical-neighborhood aggregation with first-moment graph smoothing**. The approximation to the full Wasserstein–KL proximal solution remains unvalidated.

Thank you for the response and experiments. I will hold my score of 3 (weak accept) as I believe that is the state of the current paper.

Thank you for your response and positive assessment of our work. We sincerely appreciate your time and constructive feedback throughout the review process.

Reviewer_n8HDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Policy Laplacian Trace (PLT), a way to regularize policy updates using the learner's own training history. The authors hypothesize that the policies visited during training lie on a locally low-dimensional manifold in Wasserstein space. At each step the method stores past (state, policy distribution) pairs, retrieves the k most similar past states, and pulls the current policy toward a weighted combination of those stored policies. This is done through a Wasserstein-Dirichlet energy plus a KL-proximal term. For RL, the authors compute an "aligned" reference policy with Sinkhorn transport and use it inside the PPO/MAPPO clipped objective. For language models, they use a cheaper proxy (distance between embedding means) and a gated mixture of the model output and a retrieved memory distribution. Experiments cover PPO on Atari and MuJoCo, MAPPO on StarCraft II, small GPT-style models (2.5M to 124M parameters) on synthetic stress tests, and one Qwen3-4B recall experiment. The paper also gives some theory (well-posedness and energy descent of the proximal step, in the appendix).

Critical strengths
  1. The idea of using a state-conditioned history of one's own policies as a regularizer is interesting and different from the usual single-step trust region. The ablation on simpler history-aware baselines (parameter EMA and output EMA, both negative on CarRacing) is a useful check that the gain is not just generic smoothing, and the abstract is honest about overhead (about 3.6% extra step time for Qwen3-4B).

  2. The RL evidence is broad: Atari, MuJoCo and StarCraft II, five seeds, with learning curves showing both final-return and some sample-efficiency gains. The implementation is amortized (cache the aligned policy once per PPO iteration), so the extra cost appears manageable.

Critical weaknesses
  1. The method that is actually run is quite far from the method that is motivated. The Wasserstein-manifold story (Hypothesis 1, Dirichlet energy, barycenters) is an assumption backed by one diagnostic (effective dimension of about 46 vs roughly 10^6 parameters), but the practical RL version is a kNN-weighted mixture of stored action distributions, a Sinkhorn step, a refit to a Gaussian, and then replacing pi_old in the PPO ratio with pi_align. In the LLM version, W2 is replaced by the squared distance between mean token embeddings, which is not a Wasserstein distance and cannot tell apart two very different distributions with the same mean, followed by a learned gate. So it is hard to say whether any gain comes from Wasserstein geometry, or from retrieval, or just from changing the PPO anchor. The theory (Lemma H.8, in the appendix) applies to the idealized proximal problem, not to these approximations. I also could not find, in the main text, an explanation of why anchoring on a mixture of old, stale policies should not hurt when the policy is improving quickly.

  2. The comparisons are too narrow to support the strength of the claims. The only history-aware baselines are the two EMA variants, and only on CarRacing. There is no comparison to standard KL- or trust-region-regularized PPO variants, to PPO with tuned hyperparameters, or to episodic-memory or continual-learning style regularizers, even though the paper motivates PLT with forgetting. The language-model tasks are small synthetic stress tests on models up to 124M parameters where absolute accuracies on the multi-category task are low (roughly 6 to 12%), and several relative gains are reported from near-zero baselines (+1301%, +infinity). The one realistic-scale experiment (Qwen3-4B, 3 seeds, +4.83% on a single synthetic recall task) is thin. Baseline variance in the RL results also looks large in places, so with five seeds some of the improvements may be within noise.

Questions for authors
  1. In the RL implementation, how much of the improvement remains if you keep the PPO objective unchanged and only use the retrieved historical policy as a plain KL penalty? In other words, which part of the pipeline (retrieval, transport, or the changed ratio) is responsible for the gain?

  2. How were k, the temperature, the buffer capacity and rho tuned for PLT, and were the PPO baselines given an equal tuning budget?

  3. How sensitive is the method to stale buffer entries? As the policy improves, older entries describe worse behavior, so what prevents the anchor from slowing learning, and does the buffer eviction rule matter?

  4. For the LLM version, can you show a case where the embedding-mean proxy disagrees with the actual W2 distance, and how often does that happen? Also, is the memory (and gate) used at inference time, or only during training?

  5. Could you run PLT against at least one standard baseline for catastrophic forgetting or policy drift (for example, KL-regularized PPO, or an episodic-memory regularizer) on the same tasks? This is a suggestion for strengthening the paper rather than a requirement for my assessment.

  6. Could you test the language-model claims on one established public benchmark rather than only synthetic tasks? Again a suggestion only.

4 comments
AuthorsOpenReview ↗

Response for Weaknesses

Thank you for highlighting the theory–implementation gap and the need for stronger empirical validation.

W1. Gap Between Theory and Implementation

PLT uses state-conditioned historical policies as references beyond the latest rollout policy: constraining consecutive updates alone does not directly constrain cumulative drift from earlier behavior.

Mixture versus neighbor-wise transport. Eqs. (19)–(21) retrieve historical policies qjq_j with weights wj≥0w_j \geq 0 and ∑jwj=1\sum_j w_j = 1. The original Atari implementation transports to their weighted mixture; the motivating Laplacian formulation instead sums per-neighbor transport costs. Writing CεC_\varepsilon for the implemented entropic transport cost, these objectives are

E1(p)=Cε(p,∑jwjqj),E_1(p) = C_\varepsilon(p, \sum_j w_j q_j),

E2(p)=∑jwjCε(p,qj).E_2(p) = \sum_j w_j C_\varepsilon(p, q_j).

We directly compare them by replacing E1E_1 with E2E_2, retaining retrieval, neighbor weighting, semantic action cost, and anchored clipping.

Both variants use DonkeyKong-v5, 3 training seeds, and 30M steps. Late return averages six evaluations at logged steps 27–29.5M. Normalized AUC is the trapezoidal return integral over logged steps 0–29.5M, divided by 29.5M. Peak return is the highest scheduled evaluation mean; its SD is across five evaluation episodes.

ConfigurationLate return ↑Normalized AUC ↑Peak return ± episode SD ↑
PLT with mixture transport3603.332802.374220 ± 74.83
PLT with neighbor-wise transport4636.673737.634820 ± 222.71

Neighbor-wise transport improves late return by 28.68%, normalized AUC by 33.37%, and peak return by 600 points, supporting this construction in the tested setting. We will distinguish this ablation from the original mixture-based results. Episode-level SD does not establish significance across training seeds.

PPO interface. The implemented importance ratio remains

rθ(a,s)=πθ(a∣s)πold(a∣s).r_\theta(a,s) = \frac{\pi_\theta(a \mid s)}{\pi_{old}(a \mid s)}.

The aligned historical reference shifts the clipping interval; it does not replace the numerator. We will correct the description around Eq. (25) and Algorithm 3, and distinguish finite-step entropic alignment from an exact Wasserstein proximal mapping.

W2. Baselines, Benchmark Coverage, and Statistical Significance

We added PPO with rollout-policy KL regularization, PPO with retrieved-history KL regularization, and a public WikiText-2 benchmark, plus controls for retrieval, entry age, and transport aggregation. Atari additions use DonkeyKong-v5, 3 training seeds, and 30M steps; WikiText-2 uses three paired GPT-nano training seeds.

Baseline comparisons. On DonkeyKong, replacing retrieved-history KL regularization with transport alignment increases peak return from 4100 to 4300. Adding anchored clipping yields 4220, below PPO (4480) and PPO with rollout-policy KL regularization (4560). Transport alignment therefore helps in this comparison, but the original full PLT configuration does not outperform these PPO baselines.

On WikiText-2, frozen-memory inference improves PLT's perplexity, but both PLT modes remain worse than GPT-nano:

ConfigurationTest perplexity ↓
GPT-nano655.61 ± 6.77
GPT-nano + PLT, inference retrieval disabled6366.85 ± 440.89
GPT-nano + PLT, frozen-memory inference enabled2054.95 ± 11.52

Values are mean ± sample SD across three paired training seeds. Both PLT inference modes use the same trained models.

Component ablations. Reference PLT combines state-conditioned retrieval, mixture transport, and anchored clipping. Using the metrics defined in W1:

ConfigurationLate return ↑Normalized AUC ↑
PLT3603.332802.37
PLT + entry-age limit3186.672455.93
PLT with age-matched random retrieval190.00182.37
PLT with neighbor-wise transport4636.673737.63

The age limit excludes entries older than eight PPO iterations and does not improve performance. Random retrieval replaces state-selected neighbors with random entries matched by insertion iteration while retaining the original weights; its sharp decline supports state-conditioned neighbor selection. Neighbor-wise transport replaces transport to the mixture with weighted per-neighbor costs and improves both late return and AUC.

These results identify component effects and limitations within the tested settings, not general superiority across tasks or training seeds. Return and perplexity do not directly measure retention, so we do not use them as evidence that PLT prevents catastrophic forgetting.

AuthorsOpenReview ↗

Response for Q1 - Q3

Q1&Q5 Component Ablations and Comparisons with KL and Episodic Memory Regularization Baselines

As reported in W2, we added PPO with KL regularization toward the rollout policy and PPO with KL regularization toward the retrieved historical mixture. Together with the transport and retrieval ablations in W1–W2, these comparisons separate reference construction, update rule, and neighbor selection.

Reference construction and update rule. Replacing the retrieved mixture with a semantic transport-aligned reference, while retaining PPO-Clip and the KL penalty weight, raises peak return from 4100 to 4300. Replacing this KL regularization with anchored clipping lowers the peak to 4220. Thus, the observed improvement supports transport-based reference construction, but not an additional benefit from anchored clipping. PPO with rollout-policy KL regularization achieves 4560, exceeding both the original full PLT configuration and standard PPO (4480). See W2 for the baseline comparison and W1 for the implemented importance ratio and clipping mechanism.

Ground cost. An additional comparison holds anchored clipping fixed and changes only the transport cost:

ConfigurationPeak return ± episode SD ↑
PLT with flat 0/1 action cost4060 ± 300.67
PLT with semantic action cost4220 ± 74.83

The semantic cost increases the observed peak by 160 points. Both runs use DonkeyKong-v5,3 training seeds, and 30M steps; SD is across five evaluation episodes at the peak checkpoint.

Retrieval and transport aggregation. The age-matched random-retrieval control in W2 sharply reduces performance despite retaining neighbor weights, supporting state-conditioned selection. The neighbor-wise transport comparison in W1 improves both late return and normalized AUC over transport to the historical mixture. These controls provide evidence beyond peak-return comparisons.

Scope of the evidence. The added PPO baselines address regularization and performance, not forgetting directly. These single-seed comparisons neither establish significance across training seeds nor quantify retention or cumulative policy drift. We will distinguish these claims and avoid treating improved return as evidence that PLT prevents catastrophic forgetting.

Q2. Parameter tuned in PPO and its baseline

We used the default PPO configurations from the Xuance library and kept the shared PPO hyperparameters identical for the baseline and PPO+PLT. We did not independently retune those parameters for either method.

Additional exploration was limited to PLT-specific parameters. For example, Appendix F.3 evaluates neighborhood sizes k∈{5,10,20,50}k\in\{5,10,20,50\} on Atari Alien. The comparison therefore measures the effect of adding PLT to a fixed PPO configuration; it is not an equal-budget hyperparameter search over both methods. Xuance library

Q3. Sensitivity to Stale Historical Entries

We evaluate PLT + entry-age limit, retaining the FIFO capacity of 8,192, write rule, and k=20k=20, while excluding entries older than eight PPO iterations before retrieval.

Intervention check and results. The final logged maximum and mean neighbor ages are 8 and 4.50 PPO iterations, respectively, confirming that the cutoff is active at that checkpoint; these are not full-run averages. As reported in W2, the age limit reduces late return from 3603.33 to 3186.67 (−11.56%) and normalized AUC from 2802.37 to 2455.93 (−12.36%). Peak return also decreases, from 4220 ± 74.83 to 4080 ± 256.12. Metric definitions and the episode-level interpretation of SD are provided in W1.

Restricting access to older entries therefore does not improve performance in this setting, consistent with removing useful historical information. However, age is not a measure of policy quality: this single-seed experiment tests an age-eligibility rule, not robustness to arbitrary low-quality policies or the relative merits of FIFO and alternative eviction policies. We will make these distinctions explicit.

AuthorsOpenReview ↗

Response for Q4 - Q6

Q4.1 Discrepancies Between the Embedding-Mean Proxy and W2W_2

We evaluated 72 WikiText-2 validation positions with memory hits, 24 per seed, comparing squared embedding-mean distance with exact transport on the same truncated token support after renormalization.

Validation token indexSquared embedding-mean distanceExact W22W_2^2 on truncated support
14,5900.062350.18217
109,8920.071960.15894

The proxy ranks position 14,590 as closer to its retrieved history, whereas exact transport ranks position 109,892 as closer. Across seeds, within-seed Spearman correlations range from 0.899 to 0.959, with 83 of 828 position pairs reversing order. Thus, strong rank correlation does not imply equivalence: the mean proxy can miss distributional differences beyond the first moment.

The selected support retains only 17.82% of historical probability mass on average. These findings concern truncated, renormalized distributions and do not establish approximation accuracy over the full vocabulary.

Q4.2 Memory and Gate Usage During Training and Inference

Memory retrieval and gating are used during training in all three corrected WikiText-2 runs. At inference, we evaluate two settings. Our primary evaluation disables retrieval, while an additional evaluation enables retrieval and gated fusion using a frozen memory. Both settings use the same model weights and test tokens. In the additional setting, memory writes are disabled, and the inference gate receives no target tokens.

As reported in W2, frozen-memory inference reduces mean perplexity from 6366.85 to 2054.95 (approximately 68%), with improvement in all three seeds, but remains worse than GPT-nano (655.61). This paired comparison measures the combined effect of retrieval and gating, not their separate contributions.

The inference gate is distinct from the target-dependent training hardness gate in Eqs. (36)–(38). We will explicitly separate the training and inference protocols.

Q6. Evaluation on an Established Public Language-Modeling Benchmark

We added a WikiText-2 comparison between GPT-nano and GPT-nano + PLT across three paired training seeds, matching initialization, training batches, and training budget.

As reported in W2, “Baseline comparisons,” mean test perplexity is 655.61 for GPT-nano, 6366.85 for PLT without inference retrieval, and 2054.95 for PLT with frozen-memory inference. Both PLT modes underperform GPT-nano in every seed. The same-checkpoint inference comparison and protocol are detailed in Q4.2.

Thus, inference retrieval improves PLT's own performance but does not establish an advantage over GPT-nano. We report this negative result explicitly and limit our language-model performance claims accordingly.

Dear Reviewer n8HD,

Thank you very much for your constructive feedback and thoughtful questions. Following your suggestions, we added the requested component ablations, baseline comparisons, proxy diagnostics, and WikiText-2 experiments, and revised our claims to reflect the exact scope of our findings.

With the discussion window closing soon, we wanted to gently check if our clarifications resolve your questions or if anything remains unclear. Thank you again for your time and effort in reviewing our work!

Best

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Author Discussion Summary for the Organizing Committee

Submission #24207 · Laplacian Flows for Policy Learning

Dear Organizing Committee,

We posted point-by-point replies to all three reviewers. RC9S reaffirmed weak accept (3). RFGK retained their rating of 2, found the neighbor-wise ablation helpful, and asked whether simpler geometric penalties could yield similar gains. We posted a new experiment in response. n8HD has yet to post a follow-up.

RL component ablations. On DonkeyKong-v5, with retrieval, PPO-Clip, and the KL coefficient fixed, transport alignment raised peak return from 4,100 to 4,300. Switching to anchored clipping lowered it to 4,220, compared with 4,560 for rollout-policy KL. In a separate control, changing only the transport aggregation from a historical mixture to per-neighbor costs improved late return by 28.7% and normalized AUC by 33.4%. These results support retaining per-neighbor transport and revisiting anchored clipping. A matched multi-seed comparison with rollout-policy KL remains open.

New comparison for RFGK. In a single-step reward-noise experiment with shared history and policy candidates, Wasserstein achieved the lowest utility regret (0.0599) and highest true reward (0.7619) across 128 test instances. Its regret was 27.2% lower than tuned RBF-MMD (0.0823) and below the strongest moment-based penalty (0.0777).

Revised claims. We will restrict H.13–H.14 to an exact Wasserstein proximal map with fixed references and weights on a Hilbert-isometric convex policy family. The language-model gate smooths embedding means. On WikiText-2, frozen-memory retrieval improved PLT perplexity from 6,366.85 to 2,054.95, while GPT-nano achieved 655.61. Behavioral retention remains a motivation pending direct measurement.

Thank you for considering these additions.

AuthorsOpenReview ↗

Addendum to the Global Response

Additional Update: Geometric Regularization in Actual Policy Training

Following RFGK’s remaining concern about the synthetic setting, we conducted full PPO adaptation experiments on MuJoCo Reacher-v5 under actuator perturbation and reward-estimation noise, comparing independently tuned regularizers under shared historical information, retrieval, neighbor weights, and training conditions.

Neighbor-weighted Wasserstein achieved the highest mean final return (−10.762 ± 1.033) and full-training AUC, outperforming historical forward KL and multiscale RBF-MMD. These results extend our synthetic findings to actual policy optimization and provide controlled evidence for the additional value of Wasserstein geometry beyond historical information alone.

Following these additional experiments and theoretical clarifications, RFGK acknowledged the strengthened evidence and raised their rating from 2 to 3 (weak accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.