acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing Independent Learning in General-Sum Self-Play via Coupling-Adaptive Regularization

Abstract

Independent policy updates in general-sum self-play offer computational efficiency but introduce severe non-stationarity. To guarantee joint objective improvement, existing theory bounds the induced independent-to-joint objective gap and constrains policy updates. The dependence of these bounds on the total agent count can make permissible steps vanish in large systems. We derive a bound on this gap in which each agent's update is weighted by its state-dependent coupling with the others. Under interaction sparsity, the coupling coefficient for each agent remains as the population grows. Building on this analysis, we introduce Coupling-Adaptive Regularized Policy Optimization (CARPO). CARPO uses a vector critic to estimate local coupling and adapts each agent's trust region, tightening updates in highly coupled states and relaxing them in loosely coupled ones. Empirical evaluations range from a tabular game to large-scale autonomous driving. By restricting updates during critical interactions, CARPO induces a “safe hesitation” phase that prevents premature convergence in a coordination trap. Its state-dependent adaptation outperforms the tested static clipping choices across coordination levels. In autonomous driving, CARPO maintains the rapid initial learning of independent updates while surpassing the final performance of the joint-update baseline.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, MARL, Self-play RL

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TaBAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper bounds the gap between independent and joint surrogate objectives in general-sum self-play using a state-dependent coupling coefficient instead of the usual (N−1)A_max factor, which stays O(1) under a distance-decay assumption. Based on this, CARPO estimates coupling with a vector critic and shrinks the PPO clip in highly coupled states. Experiments on the Climbing Game, LBF, and PufferDrive (10k WOMD scenarios, 10 seeds) show CARPO reaching 95.9% success versus 93.5% for GS-CoPPO and 92.1% for IPPO.

Critical strengths
  1. Replacing the (N−1)A_max factor with a variance-based local coupling term is simple and correct, and the CACC study shows it tracks the true gap while the prior bound blows up with N.

  2. The evaluation is broad and transparent, scaling from a tabular game to large-scale driving with consistent gains in all four PufferDrive subsets, disclosed baseline tuning, and reported compute cost.

Critical weaknesses
  1. CARPO only varies the clip within [0.15, 0.25] with a single PPO epoch, and fixed IPPO clips across that range barely change performance, so it is not clear that the +3.8 (mixed) and +8.5 (Low/High) point gains really come from state-dependent clipping rather than from the added vector critic.

  2. The residual in Thm 3.1 is only meaningful when ε = O(1/N), and the N-free bound in Lemma B.1 uses a uniform constant, so the theory supports an N-independent fixed clip but not the state-dependent allocation CARPO actually uses.

Questions for authors
  1. Can you run CARPO with the vector critic kept but the clip fixed at 0.2, and a variant that keeps each batch's distribution of ε but shuffles which samples get which value?

  2. With only one update epoch, how often is the clip actually active, and are the realized ratios in high-coupling states actually smaller?

  3. In the CACC setting where coupling can be computed exactly, how well does the learned vector critic estimate (K=4) agree with the true value?

  4. Does the vector critic share parameters with the actor or value network, and does it see the global state in PufferDrive?

4 comments

Thank you for your thoughtful review. We address your comments point by point below.

W1, Q1 & Q4 (Vector critic and clipping ablations)

CARPO only varies the clip within [0.15, 0.25] with a single PPO epoch, and fixed IPPO clips across that range barely change performance, so it is not clear that the +3.8 (mixed) and +8.5 (Low/High) point gains really come from state-dependent clipping rather than from the added vector critic.

Can you run CARPO with the vector critic kept but the clip fixed at 0.2, and a variant that keeps each batch's distribution of ε but shuffles which samples get which value?

Does the vector critic share parameters with the actor or value network, and does it see the global state in PufferDrive?

Vector critic. In PufferDrive, the actor and value network share a backbone. The vector critic has its own encoder and shares no parameters with either network. Appendix E clarifies this architecture in the revised paper. The vector critic takes the ego vehicle’s action and the same local observation used by the actor and value network. This observation includes features of nearby vehicles within a 50 m radius of the ego vehicle (Appendix J.3). Its predictions enter the PPO policy objective only through the coupling estimate that sets each sample’s clip threshold (Section 3.2 and Appendix F).

Clipping ablations. We compared all four methods on the PufferDrive mixed set and kept the auxiliary vector critic in both controls. When ϵ\epsilon is fixed at 0.20, the policy update reduces to the IPPO update with the same clip threshold: the vector critic trains on its own parameters, and its predictions no longer enter the policy objective. In the second control, we shuffled the thresholds across samples within each minibatch, preserving their distribution.

MethodSuccess rate (%, mean ± SD across 10 seeds)
IPPO92.1 ± 0.7
Vector critic, fixed ϵ=0.20\epsilon = 0.2092.2 ± 0.5
Vector critic, shuffled ϵ\epsilon91.6 ± 0.6
CARPO95.9 ± 0.3

With the vector critic retained and the threshold fixed, success was close to IPPO’s. Shuffling the thresholds within each minibatch also removed CARPO’s advantage over IPPO, indicating that which sample receives which threshold matters on the mixed set.


W2 (Population scaling and state-dependent clipping)

The residual in Thm 3.1 is only meaningful when ε = O(1/N), and the N-free bound in Lemma B.1 uses a uniform constant, so the theory supports an N-independent fixed clip but not the state-dependent allocation CARPO actually uses.

Population scaling (Lemma B.1). The general residual bound in Theorem 3.1 grows with NN for a fixed policy shift. Under the interaction sparsity condition, Lemma B.1 shows that Ci(s)≤Kloc\mathcal C_i(s)\leq K_{\mathrm{loc}} uniformly over states and directly bounds the full surrogate gap with the same NN-independent coefficient on each agent’s update. Thus, adding more distant agents need not force a smaller per-agent update budget.

State-dependent clipping (Theorem 3.1). At fixed NN, the residual in Theorem 3.1 is second order in the policy shift. For two states with the same small policy shift χi2(s)\sqrt{\chi_i^2(s)}, the first-order term Ci(s)χi2(s)\mathcal C_i(s)\sqrt{\chi_i^2(s)} is larger in the state with higher coupling, even when both states share the same upper bound KlocK_{\mathrm{loc}} on coupling. For the same first-order error budget, the bound therefore permits a smaller policy shift at higher coupling. CARPO uses this ordering to set its PPO clip thresholds (Section 3.2).

Q2 (Clipping activity and realized policy ratios)

With only one update epoch, how often is the clip actually active, and are the realized ratios in high-coupling states actually smaller?

With one PPO update epoch per rollout, clipping changed the policy objective for 0.73% of samples during the first 100M environment steps and 2.09% during the next 100M. Clipping was rare late in training. After 360M steps in the original full run, only 0.024% of samples had ratios outside their clip intervals and could therefore be affected by clipping.

Within each minibatch, we ranked samples by coupling and compared the top and bottom 25%. The two groups had similar mean ∣r−1∣\lvert r-1\rvert values. In both windows, clipping changed the PPO objective more often in the top quartile than in the bottom quartile (1.69% vs. 0.15% at 0–100M steps and 4.19% vs. 0.81% at 100–200M steps).

Environment stepsCoupling groupClipping active (%)Mean ∣r−1∣\lvert r-1\rvertRatio outside clip interval (%)
0–100MAll samples0.730.04411.33
0–100MLowest quartile0.150.04180.23
0–100MHighest quartile1.690.04553.12
100–200MAll samples2.090.06394.00
100–200MLowest quartile0.810.06281.48
100–200MHighest quartile4.190.06448.14

Clipping changes a sample’s objective only when the ratio exceeds its upper bound with positive advantage or falls below its lower bound with negative advantage.


Q3 (Coupling estimation in CACC)

In the CACC setting where coupling can be computed exactly, how well does the learned vector critic estimate (K=4) agree with the true value?

With K=4K=4 action samples, the critic-based coupling estimate averaged 0.0643, about 5% below the exact mean of 0.0678. We evaluated five agents in each of 80 independent CACC test states (400 state–agent pairs). The mean absolute error against the exact coupling was 0.0234 across these pairs, about 35% of the exact mean.

We estimate the surrogate C~i\tilde{\mathcal C}_i from the critic’s marginalized advantage predictions across sampled actions. Its analytical mean was 0.0677 on the same pairs.

KKMean exact coupling Ci\mathcal C_iMean analytical surrogate C~i\tilde{\mathcal C}_iMean critic-based estimateMAE vs. exact coupling
20.06780.06770.05390.0371
40.06780.06770.06430.0234
80.06780.06770.06470.0145
160.06780.06770.06660.0102
320.06780.06770.06710.0071

Increasing KK from 4 to 32 reduced the pairwise MAE from 0.0234 to 0.0071.

Reviewer_TaBAOpenReview ↗

My problem has been solved.

I am very pleased to receive your response to my questions. I believe my concerns have been addressed, and I also see that you have revised the manuscript. In my view, this paper is suitable for publication. I have decided to support your submission—consider this a one-point boost from me.

Thank you for your feedback and support!

Reviewer_GqLPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the discrepancy between a joint surrogate for the sum of individual returns and independent per-agent surrogates under simultaneous self-play updates. It derives a coupling-weighted bound based on how an agent's action affects other agents' marginalized advantages. Under interaction-decay assumptions, the coefficient associated with each agent's update can remain bounded as the population grows. The proposed CARPO algorithm estimates a tractable coupling score with a vector critic trained on GAE targets and uses the score to adapt PPO clipping ranges. The evaluation includes an analytical vehicle-following example, a coordination game, Level-Based Foraging, and PufferDrive simulations initialized from Waymo traffic scenarios. The paper reports improved performance over several policy-optimization baselines, with the clearest gains on the mixed PufferDrive benchmark.

Critical strengths
  1. The paper addresses an important limitation of independent multi-agent learning: the appropriate update size depends on how strongly agents affect one another. Its coupling-based analysis gives this observation a precise interpretation and provides a useful basis for adapting updates to the interaction structure. I find this connection between cross-agent influence and policy regularization to be the main conceptual contribution.

  2. The proposed approach connects the analysis to a practical modification of PPO. The evaluation combines controlled coordination problems with traffic simulations based on real-world scenarios, giving the paper both explanatory and practical value. The results provide encouraging evidence that accounting for interaction strength can improve independent learning.

Critical weaknesses
  1. The paper establishes CARPO's empirical usefulness more clearly than its proposed explanation of learning stability. Its central argument connects cross-agent coupling, the mismatch between independent and joint objectives, and adaptive policy updates. However, the evidence does not fully establish how this chain explains the observed improvements. The existing comparisons support the direction of adaptation, while leaving the specific contribution of the coupling signal insufficiently isolated.

  2. The practical scope of the general-sum contribution needs clearer characterization. The evidence is strongest in coordination tasks and traffic scenarios with localized interactions, but it does not yet establish how the method's benefits depend on the alignment of agents' interests. Since coupling measures the strength of cross-agent effects without distinguishing whether those effects help or harm collective return, the effectiveness of the same update rule under stronger conflicts of interest remains less clear.

Questions for authors
  1. What specific form of learning or coordination instability does CARPO reduce in the driving experiments? Please explain how that reduction is reflected in the reported results, beyond the improvement in final task performance. This would help assess the paper's central claim of stabilizing independent learning.

  2. The static-clipping comparisons and reversed-adaptation experiment provide useful support for the method. What evidence further distinguishes the benefit of coupling-informed updates from the benefit of adaptive clipping more generally? Please clarify which observations establish that the proposed coupling signal contributes to the gains.

  3. The paper motivates local coupling as a way to avoid overly conservative update control based on the total number of agents. How should this scaling advantage be compared with earlier trust-region analyses when the return objective and its normalization are matched? Please identify which part of the improvement follows specifically from exploiting local interaction structure.

  4. How does the benefit of CARPO depend on the alignment of agents' rewards? Coupling captures the magnitude of cross-agent influence, while this influence can have different consequences under aligned and conflicting incentives. Please explain which findings support the method's effectiveness when individual interests conflict, and where its practical applicability may be limited.

  5. What aspects of the interaction structure predict when CARPO will provide its largest gains? The relative improvements vary across the driving settings, and traffic density or scene complexity need not directly measure the proposed coupling. Please explain how this variation relates to the method's motivation and what the results indicate about situations in which its advantage may be modest.

4 comments

Thank you for your thoughtful review. We address your comments point by point below.

W1 & Q2 (Contribution of the coupling signal)

The paper establishes CARPO's empirical usefulness more clearly than its proposed explanation of learning stability. Its central argument connects cross-agent coupling, the mismatch between independent and joint objectives, and adaptive policy updates. However, the evidence does not fully establish how this chain explains the observed improvements. The existing comparisons support the direction of adaptation, while leaving the specific contribution of the coupling signal insufficiently isolated.

The static-clipping comparisons and reversed-adaptation experiment provide useful support for the method. What evidence further distinguishes the benefit of coupling-informed updates from the benefit of adaptive clipping more generally? Please clarify which observations establish that the proposed coupling signal contributes to the gains.

We added a shuffled-threshold control on the PufferDrive mixed set. We computed the thresholds as in CARPO, then randomly shuffled them across samples within each minibatch, preserving their distribution.

Threshold assignmentSuccess rate (%, mean ± SD across 10 seeds)
Coupling-based (CARPO)95.9 ± 0.3
Shuffled within minibatch91.6 ± 0.6

Shuffling preserved the threshold distribution but reduced success by 4.3 percentage points, indicating that the coupling-based assignment matters on the mixed set.

To see how coupling affects the policy during learning, consider the Stochastic Climbing Game, where we apply the analytical bound directly (Fig. 2(b,c)). When early miscoordination penalties drive IPPO’s probability of choosing action AA toward zero, high coupling limits CARPO’s updates and keeps P(A)P(A) near 0.5. The agents therefore continue to explore AA and eventually reach the optimal joint action (A,A)(A,A).


W2 & Q4 (Reward alignment)

The practical scope of the general-sum contribution needs clearer characterization. The evidence is strongest in coordination tasks and traffic scenarios with localized interactions, but it does not yet establish how the method's benefits depend on the alignment of agents' interests. Since coupling measures the strength of cross-agent effects without distinguishing whether those effects help or harm collective return, the effectiveness of the same update rule under stronger conflicts of interest remains less clear.

How does the benefit of CARPO depend on the alignment of agents' rewards? Coupling captures the magnitude of cross-agent influence, while this influence can have different consequences under aligned and conflicting incentives. Please explain which findings support the method's effectiveness when individual interests conflict, and where its practical applicability may be limited.

Our joint objective is the sum of individual returns, J(π)=∑iJi(π)J(\boldsymbol{\pi})=\sum_i J_i(\boldsymbol{\pi}) (Section 2.2). When incentives conflict, losses to other agents can outweigh one agent’s gain. Theorem 3.1 bounds the gap between the independent and joint surrogates using the sensitivity of other agents’ marginalized advantages to the updating agent’s action. Strong coupling therefore motivates a tighter clip whether the effects are beneficial or harmful.

CARPO outperforms IPPO both in the Stochastic Climbing Game, where agents share a reward, and on the PufferDrive mixed set (95.9% vs. 92.1% success), where vehicles pursue individual destinations and coordinate at bottlenecks (Fig. 5(c)). In a strictly zero-sum two-player game, the sum of returns is zero for every joint policy, so our bound does not establish progress toward a minimax solution.


Q1 (Learning stability in PufferDrive)

What specific form of learning or coordination instability does CARPO reduce in the driving experiments? Please explain how that reduction is reflected in the reported results, beyond the improvement in final task performance. This would help assess the paper's central claim of stabilizing independent learning.

CARPO targets interference between simultaneous policy updates in strongly coupled situations.

Lower variability across training runs. We averaged each run’s success rate over the final 10% of training. Across 10 seeds, the standard deviation was lower for CARPO than for IPPO on all four driving subsets and on the mixed set (Table 1). On the mixed set, it was 0.3 percentage points for CARPO and 0.7 for IPPO.

High coupling at traffic bottlenecks. In the interchange shown in Fig. 5(c), high estimated coupling is concentrated at bottlenecks where one vehicle’s actions can affect several others’ progress. CARPO therefore assigns tighter clip thresholds at these locations, where vehicles need to coordinate.

Q3 (Scaling under matched normalization)

The paper motivates local coupling as a way to avoid overly conservative update control based on the total number of agents. How should this scaling advantage be compared with earlier trust-region analyses when the return objective and its normalization are matched? Please identify which part of the improvement follows specifically from exploiting local interaction structure.

Return normalization. Dividing the sum-of-returns objective by NN gives the team-return scale under shared rewards. Under this normalization, both first-order terms average over the NN agent updates. Within that average, the global term weights each agent’s policy shift by (N−1)Amax⁡(N-1)A_{\max}, while our term uses the state-dependent coupling Ci(s)\mathcal C_i(s). Figure 1 compares both terms with ΔL/N\Delta\mathcal L/N for the same policy update and old-policy state distribution, using a sampled maximum advantage for the global term (Appendix G.2).

Advantage centering. An agent’s change in action probabilities sums to zero, so we can subtract a baseline independent of its action before bounding its effect on another agent’s advantage. Only the variation in that advantage across the updating agent’s actions contributes. For example, if the advantage is large but constant across those actions, the pair contributes zero to the first-order term. This refinement follows from centering alone and does not require interaction sparsity.

Interaction sparsity and population scaling. Under the interaction sparsity condition, pairwise influence decays exponentially with distance, while the number of agents at each distance grows at most polynomially. The sum of these influences therefore stays bounded as agents are added. The coefficient for each update is then bounded by Kloc=O(1)K_{\mathrm{loc}}=O(1) independently of NN. For the same first-order error budget per update, the global coefficient gives an update range proportional to 1/(N−1)1/(N-1) at fixed Amax⁡A_{\max}. Under interaction sparsity, adding distant agents need not shrink the range allowed by the coupling-based bound (Fig. 1(c)).


Q5 (Interaction structure and performance gains)

What aspects of the interaction structure predict when CARPO will provide its largest gains? The relative improvements vary across the driving settings, and traffic density or scene complexity need not directly measure the proposed coupling. Please explain how this variation relates to the method's motivation and what the results indicate about situations in which its advantage may be modest.

We expect CARPO to gain most when strongly and weakly coupled decisions occur within the same update. A fixed threshold must balance limiting the surrogate gap in strongly coupled samples against allowing larger updates in weakly coupled samples, while CARPO adjusts the threshold to each sample’s coupling. When coupling is similar across decisions, a well-chosen fixed threshold can already suit most samples, leaving less room for adaptation.

Thank you to the authors for their helpful responses and clarifications. I appreciate the additional explanations. After considering the responses, my overall assessment remains unchanged, and I will maintain my positive rating.

Thank you for taking the time to consider our responses. We appreciate your thoughtful feedback and continued support.

Reviewer_eNHkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper has quite a nice idea; we should vary the size of the clip coefficient in PPO-like methods based on how bad the approximation of each agent learning independently is. They provide a theorem showing that there is a seemingly tractable bound that can be used to estimate how one should shrink the clipping term.

Critical strengths
  • I appreciate the use of the driving simulator to show scaled up version of the experimental results.
  • The idea of variable clipping, based on joint advantage, seems like quite a nice idea and is motivated by an interesting bound.
Critical weaknesses
  • The bound seems like it could be become quite large. For some reason that I don't fully grok, most of the analysis of the bound is on a CACC task that is not referenced in the main paper but primarily in the appendix?
  • I really do think the paper needs plots of how epsilon is varied by CARPO in the environments. Without it, it's hard to understand exactly what it is doing or whether the machinery, which is interesting, could simply be replaced by some kind of simple epsilon schedule (even if the theory would be used to motivate such a thing) rather than this more complicated mechanism.

Small nits:

  • You refer to the "independent to joint objective gap" multiple times before defining it or citing a source, particularly in the intro.
Questions for authors
  • I do not understand the implications of 3.1 in the following sense. The second term seems exponential in N, which is worse than the linear factor used previously. How does the remark about scalability make sense in light of this?
  • Can you describe the tuning procedure used for PPO? CARPO adds additional hyperparameters, so to make the comparison fair, you really should try to give similar tuning to the baselines as the tuning used to pick the CARPO hparams.
  • Maybe I missed it but is there a plot of how CARPO evolves the clip term in LBF? It would have been really valuable to see how much control is executed there.
  • It seems like for the number of iterations used in pufferdrive, that the initial value of the EMA term has not really shifted much over the course of training. This makes it seem possible that most of the effect is coming from the EMA over the clip parameter.
2 comments

Thank you for your thoughtful review. We address your comments point by point below.

W1 (CACC and the coupling bound)

The bound seems like it could be become quite large. For some reason that I don't fully grok, most of the analysis of the bound is on a CACC task that is not referenced in the main paper but primarily in the appendix?

When many agents interact strongly, an update evaluated with the other policies held fixed can have a different effect when those policies change simultaneously. The independent-to-joint surrogate gap can therefore be large, in which case any valid upper bound must also be large. In CACC with 100 vehicles, increasing the interaction radius from 1 to 16 raises the mean normalized gap from 0.0095 to 0.0299 and the coupling-dependent first-order term from 0.0169 to 0.0467 (Appendix G, Table 2). Thus, at the same population size, the larger first-order term reflects a larger observed gap as interactions strengthen.

Given the main-text space limit, we presented the CACC results in Fig. 1(b,c) and described the task in Appendix G. We now introduce CACC as a vehicle-following task alongside Fig. 1 in the revised introduction.


W2 & Q3 (Clip thresholds in LBF)

I really do think the paper needs plots of how epsilon is varied by CARPO in the environments. Without it, it's hard to understand exactly what it is doing or whether the machinery, which is interesting, could simply be replaced by some kind of simple epsilon schedule (even if the theory would be used to motivate such a thing) rather than this more complicated mechanism.

Maybe I missed it but is there a plot of how CARPO evolves the clip term in LBF? It would have been really valuable to see how much control is executed there.

A schedule based on training progress assigns one clip threshold to all samples in an update. CARPO uses estimated coupling to decide which samples should receive tighter thresholds. For the same policy shift, a sample with higher coupling contributes more to the first-order gap bound, which motivates tighter clipping on that sample. Figure 5(c) maps estimated coupling within a single driving scene, with the highest values at traffic bottlenecks (see also Appendix J.6, Fig. 16).

We added Fig. 11 to Appendix I.2 of the revised paper to show how both the mean threshold and its within-batch standard deviation evolve in LBF. In both tasks, the mean threshold rises early, falls, and recovers later in training. The within-batch standard deviation remains substantial, showing that thresholds continue to vary across samples. A time schedule could reproduce the mean trend while missing the coupling-dependent differences between samples in the same update.


Nit (Definition of the surrogate gap)

You refer to the "independent to joint objective gap" multiple times before defining it or citing a source, particularly in the intro.

We now define the gap at first use in the introduction of the revised paper.

Q1 (Scaling of the gap bound)

I do not understand the implications of 3.1 in the following sense. The second term seems exponential in N, which is worse than the linear factor used previously. How does the remark about scalability make sense in light of this?

At fixed NN, the residual in Theorem 3.1 is second order in the policy shift, and its coefficients grow with NN.

Under the interaction sparsity condition, each agent’s influence on other agents’ advantages decays with distance fast enough that its total influence stays bounded as NN grows. Lemma B.1 then bounds the full surrogate gap by KlocK_{\mathrm{loc}} times the sum of the individual policy shifts. The scalability remark refers to this bound: its coefficient KlocK_{\mathrm{loc}} is independent of NN.


Q2 (PPO tuning procedure)

Can you describe the tuning procedure used for PPO? CARPO adds additional hyperparameters, so to make the comparison fair, you really should try to give similar tuning to the baselines as the tuning used to pick the CARPO hparams.

We used PufferDrive’s PPO implementation for both IPPO and CARPO and tuned hyperparameters only on the mixed scenario set. For IPPO, we compared three fixed clip values (0.15, the default 0.20, and 0.25) with the same training budget per run. Mean success rates over the final 10% of training were 91.9%, 92.1%, and 91.1%, respectively. We selected 0.20 and used it across all four subsets.

For CARPO, we tuned only the neighbor count MM and the action sample count KK on the mixed scenario set, testing {2,4,8}\{2,4,8\} for each separately. We used the selected values M=K=4M=K=4 across all four subsets, with the EMA decay fixed at 0.99 and the clip range fixed at [0.15,0.25][0.15, 0.25].


Q4 (EMA initialization and normalization)

It seems like for the number of iterations used in pufferdrive, that the initial value of the EMA term has not really shifted much over the course of training. This makes it seem possible that most of the effect is coming from the EMA over the clip parameter.

In PufferDrive, the EMA tracks mean coupling, which serves as the reference for each sample’s clip threshold. We initialize it with the first valid minibatch mean and update it on each subsequent valid optimization minibatch. With about 288 minibatches per rollout and a decay of 0.99, the initial mean has only 0.99287≈5.6%0.99^{287} \approx 5.6\% of the weight after one rollout. Replacing the EMA with the current minibatch mean lowered success on the mixed set from 95.9% to 93.7%, still above IPPO’s 92.1% (Appendix J.5, Table 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.