- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows how SAM improves the robustness of PPO to action noise. The authors examine the role of training scale and endpoint reward flatness. They use scale history replay to compare different scale histories and mean updaters. They analyze feedback from the mean gradient to the policy scale. Proposition 1 describes one part of this feedback. The authors test the feedback through routing experiments and explicit scale controls. Finally, they compare PPO and Mean SAM at a fixed training scale. They measure reward flatness, parameter perturbation dose and action displacement. Main contribution is the experimental analysis of training scale in SAM with PPO. The scale history replay provides useful evidence about the role of training scale. The paper also introduces RFD and provides controls for reward flatness measurements. However, the theoretical analysis does not fully explain how the native SAM updates produce larger training scales and better robustness. The experiments also do not show whether the main conclusions remain similar under other SAM radius values and fixed training scales.
Critical strengths
- Convincing scale history replay analysis (Section 3.2, Figure 2, Appendix C.1, Tables 6 and 7) The scale history replay is the strongest part of the paper. The authors use PPO and Mean SAM with different scale histories. Using the SAM scale history improves action noise robustness under both mean updaters. This improvement is observed in all four environments. Table 7 reports positive Action AUC differences in all eight comparisons. The results support the important role of training scale in robustness.
- Detailed experiments and controls (Sections 3.3 and 4, pp. 4 to 6, Appendices B, C and D) The paper provides several experiments to support its findings. The fixed scale experiments show how scale level affects robustness. The routing experiments examine feedback from the mean gradient to the scale. The entropy and RFD controls show that robustness can improve without the full SAM update. The authors also report 15 training seeds and additional results in the appendices. These experiments provide useful evidence for the main claims.
- Analysis of reward flatness (Section 5, Tables 3 and 4, pp. 7 and 8, Appendix E.2, Tables 12 to 14, pp. 30.) The authors compare PPO and Mean SAM using a common parameter RMS dose. The measured flatness advantage becomes smaller across the four tasks, although Ant still shows a positive difference. The authors then compare return retention at similar action displacements. The results show no consistent advantage across tasks under this matching method.
Critical weaknesses
- Limited connection between theory and training (Sections 4.1 and 4.2, pp. 5 and 6, Appendix A.1, pp. 14 to 16, Appendix B.2, pp. 19 and 20) Proposition 1 analyzes one part of the feedback from the mean gradient to the policy scale. This part has a nonnegative total effect on the entropy direction. However, the total feedback contains other terms. The routing experiments test the full feedback from the mean to the scale. They do not measure the contribution of each term during native SAM training. The routing experiments also use gradient norm controls that differ from the native SAM update. The authors explain these limitations in the appendix. However, it remains unclear how much the result in Proposition 1 explains the scale changes observed during training. The theory supports the proposed explanation, but it does not fully explain the observed robustness gains.
- Limited analysis of scale history (Section 3.2, pp 4, Section 3.3, pp 5, Appendix B.2, pp 19, Appendix C.1, pp 23 to 25) The replay experiments use a different training procedure from native SAM. The scale follows a recorded history and remains fixed during each update cycle. Mean SAM also perturbs only the mean parameters. The authors report differences between self replay and native training in Appendix C.1. Therefore, the replay results support the role of scale history within this protocol, but they do not fully explain the robustness gains of native SAM.
- Limited analysis of the SAM perturbation radius (Section 3, pp 3, Section 4.1, pp 5, Appendix B.1, Table 5, pp 18) The main experiments use a SAM perturbation radius of 0.008. The paper does not report whether the main results remain similar at other radius values. The radius affects the SAM update and the feedback analyzed in Proposition 1. The results may depend on the selected radius. An additional experiment with another radius would help check whether the scale history remains more important than the mean updater.
- Fixed scale comparison uses only one scale value (Section 3.3, Figure 3, pp 5, Section 5.2, Table 3, pp 7, Appendix C.2, Table 8, pp 26) Main reward flatness experiments compare PPO and Mean SAM with the scale fixed at 0.20. However, Figure 3 shows that different training scales produce different returns and robustness results. The paper does not show whether the same reward flatness results hold at other fixed scale values. Testing another scale value would help check whether the results depend on the selected training scale.
- Limitations of action displacement matching (Section 5.3, Table 4, page 8, Appendix B.3, Appendix E.2, Tables 13 ) The matching method compares the average action displacement on shared states. It does not compare the direction of action changes or the resulting trajectories. The matched points also do not have the same parameter perturbation dose. The authors mention these limitations and provide additional tests. The results show no consistent advantage in return retention under this matching method. However, they do not establish that the two policies behave the same under other perturbations.
Questions for authors
o How was the SAM perturbation radius of 0.008 selected? Do the scale history replay results remain similar at another radius value? o Proposition 1 describes one part of the scale feedback. How large is this part compared with the other feedback terms during native SAM training? o The fixed scale experiments show that scale level affects robustness. Do the timing and scale of each action dimension also affect robustness? Can the authors explain this based on the reported results?
3 comments
Response to Reviewer vivb (2/3)
3. SAM radius (W3, Q1)
Choice. We used ρ = 0.008 across all tasks, taking the HalfCheetah/Walker2d setting reported by Lee & Yoon as a reference and keeping one shared radius rather than tuning it per task.
Five-radius sweep (ρ ∈ {0.002, 0.004, 0.008, 0.016, 0.032}, n = 15; not reported in the submission). Mean final increases monotonically with ρ in all four environments. Action-AUC point estimates increase from 0.004 to 0.016 in all four; Hopper changes little between 0.002 and 0.004. At 0.032, action AUC declines in HalfCheetah and Ant alongside lower nominal returns. At 0.008, mean nominal return is highest among the five settings in HalfCheetah and Walker2d and near the highest in Ant, but not in Hopper.
Replay at a second radius (new; ρ = 0.016, seeds 1–10). The protocol is unchanged: the SAM scale history comes from SAM-PPO at ρ = 0.016, and Mean-SAM also uses 0.016.
Table G. Scale-history effect on action AUC: SAM history − PPO history under the same mean updater. Means [95% paired seed-bootstrap intervals] at ρ = 0.016; parentheses give the corresponding means at ρ = 0.008 on the same seeds.
| Env | PPO mean updater | Mean-SAM mean updater |
|---|---|---|
| HalfCheetah | +890 [+542, +1241] (+695) | +854 [+466, +1198] (+821) |
| Hopper | +397 [+319, +475] (+291) | +372 [+267, +467] (+346) |
| Walker2d | +1392 [+1177, +1631] (+961) | +1239 [+1076, +1413] (+901) |
| Ant | +494 [+413, +577] (+408) | +663 [+613, +729] (+496) |
- All eight scale-history intervals are above zero. All eight point estimates are larger than their counterparts at ρ = 0.008.
- The mean-updater effect (Mean-SAM − PPO under the same history) ranges from −330 to +63 at ρ = 0.016. Six of eight point estimates are negative; three intervals are below zero.
- At both tested radii, every scale-history effect is larger in point estimate than the absolute mean-updater effects in the same environment.
Thus, the larger effect of changing the scale history than changing the mean updater persists at a second radius within this replay protocol.
Radius and Prop 1. At a fixed parameter point, all three first-order terms are proportional to ρ. We measured their cancellation only at ρ = 0.008.
4. Another fixed scale (W4)
We compared Mean-SAM with PPO at fixed σ = 0.40 (seeds 1–10; flatness protocol of §5; not pre-registered). Mean-SAM perturbs only the mean parameters, with ρ = 0.008.
Table H. Mean-SAM − PPO at σ = 0.40, means [95% paired seed-bootstrap intervals]. The submitted σ = 0.20 comparison uses n = 15 (Table 3).
| Env | ΔA80 | Δ action AUC | ΔnAUC | ΔA80 at common parameter perturbation size |
|---|---|---|---|---|
| HalfCheetah | +0.094 [−0.011, +0.188] | −228 [−499, +24] | −0.029 [−0.060, +0.002] | +0.014 [−0.092, +0.103] |
| Hopper | +0.016 [−0.147, +0.177] | −166 [−365, −2] | +0.278 [−0.010, +0.661] | −0.070 [−0.236, +0.100] |
| Walker2d | +0.216 [+0.047, +0.382] | −146 [−379, +77] | −0.058 [−0.161, +0.013] | +0.143 [−0.030, +0.316] |
| Ant | +0.186 [+0.124, +0.250] | +72 [+34, +111] | +0.003 [−0.012, +0.020] | +0.039 [−0.027, +0.106] |
- Flatness: A80 point estimates are higher for Mean-SAM in all four environments, as at σ = 0.20. The intervals are above zero in Walker2d and Ant.
- Action-noise robustness: there is no consistent gain across environments at either scale. At σ = 0.40, Ant has a positive action-AUC difference, alongside higher nominal return (+162 [+33, +293]); its nAUC difference interval contains zero.
- Common parameter perturbation size: all four A80 difference intervals contain zero.
Three details matter for interpretation:
- Under the looser displacement filter retaining 75% of PPO's displacement distribution, Walker2d and Ant retain positive A80 differences with intervals above zero. Under the stricter 25% and 50% filters, all four intervals contain zero.
- In Hopper, 3/10 Mean-SAM seeds finish at low returns after late-training oscillations. All seeds are retained. Across all ten seeds, nominal return is lower than PPO's by −914 [−1505, −370], and action AUC is also lower. Low zero-noise returns in unstable seeds inflate their nAUC ratios.
- A80 uses 80% of each model's own grid-center return as its threshold. It measures relative return retention, not absolute perturbed performance.
Response to Reviewer vivb (3/3)
5. Limits of displacement matching (W5)
Agreed. Matching controls only the average action-displacement magnitude on shared states. It does not match action-change directions, state-wise patterns, or subsequent trajectories; matched points also need not have the same parameter perturbation size.
For the submitted fixed-σ = 0.20 comparison, the six post hoc matching rules in Appendix E.2 give the same qualitative conclusion: no consistent matched return-retention advantage for Mean-SAM across tasks. This conclusion concerns the reported scalar matching, not broader behavioral equivalence.
The separate action-noise evaluation runs whole episodes in closed loop and includes the resulting state distributions and trajectories. Fixed-scale Mean-SAM shows no consistent gain across environments there either (point 4). External random action noise differs from parameter-induced action changes, so this evaluation complements displacement matching rather than replacing it.
6. Timing and per-dimension scale (Q3)
We provide two analyses not reported in the submission, both with n = 15.
Timing. During 3M environment steps, SAM is enabled only in the first or last 0.5M or 1M steps; PPO is used outside that window. All 16 window/environment combinations have lower action-AUC point estimates than full-training SAM-PPO, with difference intervals below zero in 15.
Table I. Late − early placement of the same 1M-step SAM window: last 1M minus first 1M steps. Means [95% paired seed-bootstrap intervals].
| Env | Δ final | Δ nominal return | Δ action AUC | ΔnAUC |
|---|---|---|---|---|
| HalfCheetah | +0.053 [+0.046, +0.059] | −682 [−1463, +108] | +33 [−168, +262] | +0.063 [−0.006, +0.134] |
| Hopper | +0.077 [+0.071, +0.084] | +45 [−428, +521] | +141 [+66, +213] | +0.079 [−0.009, +0.162] |
| Walker2d | +0.041 [+0.030, +0.052] | −363 [−764, +102] | +73 [−68, +211] | +0.063 [+0.025, +0.100] |
| Ant | +0.069 [+0.065, +0.073] | −836 [−1293, −367] | +6 [−74, +81] | +0.066 [+0.046, +0.087] |
- Later placement raises final scale in all four environments, with intervals above zero. nAUC point estimates also rise in all four; its intervals are above zero in Walker2d and Ant. The action-AUC interval is above zero only in Hopper.
- Early-only SAM ends at a lower scale than full-training SAM-PPO. For example, HalfCheetah ends at 0.060 versus 0.156 for full-training SAM-PPO and 0.041 for PPO.
- For the 0.5M windows, all four late-minus-early nAUC intervals contain zero; Ant's action-AUC difference is negative (−118 [−168, −62]).
These results show that the timing of SAM-enabled training matters. Later 1M-step windows produce higher final scales alongside higher nAUC point estimates. Because the experiment switches the whole SAM update, it changes both mean and scale trajectories and does not isolate the effect of scale timing from final scale or mean-update effects.
Per-dimension scale (final checkpoints of the main batch; 23 action coordinates across four environments).
- SAM-PPO's σ is higher than PPO's in all 15 seeds for every coordinate. Coordinate-wise geometric-mean ratios across seeds range from 3.2 to 4.9.
- In log-scale, the component shared by all coordinates accounts for 92.5–97.4% of the squared change, averaged over seeds within each environment.
- Between-coordinate log-scale spread has lower point estimates in all four environments; difference intervals are below zero in Walker2d and Ant and contain zero in HalfCheetah and Hopper.
Thus, most of the observed scale increase is shared across coordinates. Replay replaces the full per-dimension history, while fixed-scale experiments assign the same σ to every coordinate. Neither comparison isolates the effect of the dimensional profile while holding the overall scale history fixed.
Planned revision. We will clarify radius selection and replay's scope in §3, routing's scope in §4, and the matching limits in §5.3 and Appendix E.2. The appendix will add the radius sweep, Tables G–I, the per-dimension analysis, and the learned-scale training comparisons in point 2 during the discussion period.
Response to Reviewer vivb (1/3)
Thank you for the detailed review. Below we address each concern with the corresponding evidence. All intervals are 95%.
Short answers
1. Prop 1 and scale changes in training (W1, Q2)
Size. Fixed-checkpoint probes along SAM-PPO runs (15 seeds × 10 checkpoints × 32 minibatches per environment, before clipping) give coordinate-sum means, ×10⁻³: cross term +51.7 to +100.0, ratio term −47.7 to −93.8, and scale term −0.45 to −0.76. The full first-order feedback is +3.5 to +10.8, or 6–14% of the cross term. Its median cosine with the measured PPO − SAM gradient difference on φ is 0.995–0.999.
Training (general response, n = 10). Removing only the cross term from SAM-PPO puts mean final σ below PPO in all four environments; adding it as extra feedback to PPO gives 1.19–1.50× SAM-PPO's scale. Removing the mean-to-scale feedback (cross term + ratio term) removes 48–55% of the difference in mean final σGM between SAM-PPO and PPO. A further 13–34% disappears after removing clipping cross-talk; 18–32% remains unattributed. These are sequential terminal contrasts that depend on removal order. The general response expands Prop 1 and corrects its net-effect interpretation.
Routing controls. We compared ψ→φ with ψ→ψ under three protocols (n = 15). Under standard routing (full clean-gradient normalization, shared clipping), final σGM differences are +0.024 / +0.068 / +0.055 / +0.075 (HalfCheetah / Hopper / Walker2d / Ant), all with intervals above zero. All 12 scale-difference intervals across the three protocols are above zero. The paper's clipping-neutral protocol has contrasts 1.5–2.6× larger than the other protocols. Protocols span different code versions and 10 vs. 50 evaluation episodes; comparisons are paired within, not across, protocols (Table F, K9WT's thread). To first order, routing transmits both cross and ratio terms, not the cross term alone.
2. Replay differs from native training (W2)
Agreed. Replay prescribes σ from a recorded history, holds it fixed within each update, and applies Mean-SAM only to the mean. It identifies history effects within this controlled 2×2 protocol, not a fraction of native SAM-PPO's robustness gain.
Appendix C.1 reports self-replay/native differences. At ρ = 0.016, Mean-SAM self-replay minus native SAM-PPO has action-AUC differences −273 / −71 / +68 / +26, with all intervals containing zero; this is not an equivalence test.
New scale-update interventions with σ learned online (n = 10):
These results connect the controlled replay finding to scale-update interventions with σ learned online.