- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper bounds the gap between independent and joint surrogate objectives in general-sum self-play using a state-dependent coupling coefficient instead of the usual (N−1)A_max factor, which stays O(1) under a distance-decay assumption. Based on this, CARPO estimates coupling with a vector critic and shrinks the PPO clip in highly coupled states. Experiments on the Climbing Game, LBF, and PufferDrive (10k WOMD scenarios, 10 seeds) show CARPO reaching 95.9% success versus 93.5% for GS-CoPPO and 92.1% for IPPO.
Critical strengths
-
Replacing the (N−1)A_max factor with a variance-based local coupling term is simple and correct, and the CACC study shows it tracks the true gap while the prior bound blows up with N.
-
The evaluation is broad and transparent, scaling from a tabular game to large-scale driving with consistent gains in all four PufferDrive subsets, disclosed baseline tuning, and reported compute cost.
Critical weaknesses
-
CARPO only varies the clip within [0.15, 0.25] with a single PPO epoch, and fixed IPPO clips across that range barely change performance, so it is not clear that the +3.8 (mixed) and +8.5 (Low/High) point gains really come from state-dependent clipping rather than from the added vector critic.
-
The residual in Thm 3.1 is only meaningful when ε = O(1/N), and the N-free bound in Lemma B.1 uses a uniform constant, so the theory supports an N-independent fixed clip but not the state-dependent allocation CARPO actually uses.
Questions for authors
-
Can you run CARPO with the vector critic kept but the clip fixed at 0.2, and a variant that keeps each batch's distribution of ε but shuffles which samples get which value?
-
With only one update epoch, how often is the clip actually active, and are the realized ratios in high-coupling states actually smaller?
-
In the CACC setting where coupling can be computed exactly, how well does the learned vector critic estimate (K=4) agree with the true value?
-
Does the vector critic share parameters with the actor or value network, and does it see the global state in PufferDrive?
4 comments
Q2 (Clipping activity and realized policy ratios)
With only one update epoch, how often is the clip actually active, and are the realized ratios in high-coupling states actually smaller?
With one PPO update epoch per rollout, clipping changed the policy objective for 0.73% of samples during the first 100M environment steps and 2.09% during the next 100M. Clipping was rare late in training. After 360M steps in the original full run, only 0.024% of samples had ratios outside their clip intervals and could therefore be affected by clipping.
Within each minibatch, we ranked samples by coupling and compared the top and bottom 25%. The two groups had similar mean values. In both windows, clipping changed the PPO objective more often in the top quartile than in the bottom quartile (1.69% vs. 0.15% at 0–100M steps and 4.19% vs. 0.81% at 100–200M steps).
| Environment steps | Coupling group | Clipping active (%) | Mean | Ratio outside clip interval (%) |
|---|---|---|---|---|
| 0–100M | All samples | 0.73 | 0.0441 | 1.33 |
| 0–100M | Lowest quartile | 0.15 | 0.0418 | 0.23 |
| 0–100M | Highest quartile | 1.69 | 0.0455 | 3.12 |
| 100–200M | All samples | 2.09 | 0.0639 | 4.00 |
| 100–200M | Lowest quartile | 0.81 | 0.0628 | 1.48 |
| 100–200M | Highest quartile | 4.19 | 0.0644 | 8.14 |
Clipping changes a sample’s objective only when the ratio exceeds its upper bound with positive advantage or falls below its lower bound with negative advantage.
Q3 (Coupling estimation in CACC)
In the CACC setting where coupling can be computed exactly, how well does the learned vector critic estimate (K=4) agree with the true value?
With action samples, the critic-based coupling estimate averaged 0.0643, about 5% below the exact mean of 0.0678. We evaluated five agents in each of 80 independent CACC test states (400 state–agent pairs). The mean absolute error against the exact coupling was 0.0234 across these pairs, about 35% of the exact mean.
We estimate the surrogate from the critic’s marginalized advantage predictions across sampled actions. Its analytical mean was 0.0677 on the same pairs.
| Mean exact coupling | Mean analytical surrogate | Mean critic-based estimate | MAE vs. exact coupling | |
|---|---|---|---|---|
| 2 | 0.0678 | 0.0677 | 0.0539 | 0.0371 |
| 4 | 0.0678 | 0.0677 | 0.0643 | 0.0234 |
| 8 | 0.0678 | 0.0677 | 0.0647 | 0.0145 |
| 16 | 0.0678 | 0.0677 | 0.0666 | 0.0102 |
| 32 | 0.0678 | 0.0677 | 0.0671 | 0.0071 |
Increasing from 4 to 32 reduced the pairwise MAE from 0.0234 to 0.0071.
My problem has been solved.
I am very pleased to receive your response to my questions. I believe my concerns have been addressed, and I also see that you have revised the manuscript. In my view, this paper is suitable for publication. I have decided to support your submission—consider this a one-point boost from me.
Thank you for your feedback and support!
Thank you for your thoughtful review. We address your comments point by point below.
W1, Q1 & Q4 (Vector critic and clipping ablations)
Vector critic. In PufferDrive, the actor and value network share a backbone. The vector critic has its own encoder and shares no parameters with either network. Appendix E clarifies this architecture in the revised paper. The vector critic takes the ego vehicle’s action and the same local observation used by the actor and value network. This observation includes features of nearby vehicles within a 50 m radius of the ego vehicle (Appendix J.3). Its predictions enter the PPO policy objective only through the coupling estimate that sets each sample’s clip threshold (Section 3.2 and Appendix F).
Clipping ablations. We compared all four methods on the PufferDrive mixed set and kept the auxiliary vector critic in both controls. When ϵ is fixed at 0.20, the policy update reduces to the IPPO update with the same clip threshold: the vector critic trains on its own parameters, and its predictions no longer enter the policy objective. In the second control, we shuffled the thresholds across samples within each minibatch, preserving their distribution.
With the vector critic retained and the threshold fixed, success was close to IPPO’s. Shuffling the thresholds within each minibatch also removed CARPO’s advantage over IPPO, indicating that which sample receives which threshold matters on the mixed set.
W2 (Population scaling and state-dependent clipping)
Population scaling (Lemma B.1). The general residual bound in Theorem 3.1 grows with N for a fixed policy shift. Under the interaction sparsity condition, Lemma B.1 shows that Ci(s)≤Kloc uniformly over states and directly bounds the full surrogate gap with the same N-independent coefficient on each agent’s update. Thus, adding more distant agents need not force a smaller per-agent update budget.
State-dependent clipping (Theorem 3.1). At fixed N, the residual in Theorem 3.1 is second order in the policy shift. For two states with the same small policy shift χi2(s), the first-order term Ci(s)χi2(s) is larger in the state with higher coupling, even when both states share the same upper bound Kloc on coupling. For the same first-order error budget, the bound therefore permits a smaller policy shift at higher coupling. CARPO uses this ordering to set its PPO clip thresholds (Section 3.2).