acceptodds
Under review as a conference paper at ICLR 2027

Beyond Endpoint Flatness: Policy Scale Dynamics Shape Robustness in SAM-PPO

Abstract

Sharpness-Aware Minimization (SAM) improves PPO robustness to action noise, and prior work links this gain to flatter endpoint reward landscapes. We study how this robustness forms and what endpoint geometry can identify in diagonal-Gaussian PPO with learnable training scale, evaluating deterministic mean policies under external action noise. Across four MuJoCo tasks, replaying complete scale histories shows that action robustness largely follows the training-scale history, while changing the mean updater has smaller, task-dependent effects. Local analysis and routing interventions further support feedback from mean perturbations into the scale update, and explicit scale controls show that this pathway can be regulated without the full SAM update. With training scale fixed, Mean-SAM produces flatter relative reward landscapes without consistent action-noise gains. Equalizing the absolute parameter perturbation dose substantially weakens this flatness advantage, and further matching the induced action displacements reveals no consistent return-retention advantage across tasks. Together, these results identify training scale as a functional pathway behind SAM-PPO robustness and show that endpoint flatness alone cannot determine how that robustness was formed.

Primary area:
reinforcement learning
Keywords:
Sharpness-Aware Minimization, Proximal Policy Optimization, Robust Reinforcement Learning, Stochastic Policy Optimization, Policy Scale

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vivbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows how SAM improves the robustness of PPO to action noise. The authors examine the role of training scale and endpoint reward flatness. They use scale history replay to compare different scale histories and mean updaters. They analyze feedback from the mean gradient to the policy scale. Proposition 1 describes one part of this feedback. The authors test the feedback through routing experiments and explicit scale controls. Finally, they compare PPO and Mean SAM at a fixed training scale. They measure reward flatness, parameter perturbation dose and action displacement. Main contribution is the experimental analysis of training scale in SAM with PPO. The scale history replay provides useful evidence about the role of training scale. The paper also introduces RFD and provides controls for reward flatness measurements. However, the theoretical analysis does not fully explain how the native SAM updates produce larger training scales and better robustness. The experiments also do not show whether the main conclusions remain similar under other SAM radius values and fixed training scales.

Critical strengths
  1. Convincing scale history replay analysis (Section 3.2, Figure 2, Appendix C.1, Tables 6 and 7) The scale history replay is the strongest part of the paper. The authors use PPO and Mean SAM with different scale histories. Using the SAM scale history improves action noise robustness under both mean updaters. This improvement is observed in all four environments. Table 7 reports positive Action AUC differences in all eight comparisons. The results support the important role of training scale in robustness.
  2. Detailed experiments and controls (Sections 3.3 and 4, pp. 4 to 6, Appendices B, C and D) The paper provides several experiments to support its findings. The fixed scale experiments show how scale level affects robustness. The routing experiments examine feedback from the mean gradient to the scale. The entropy and RFD controls show that robustness can improve without the full SAM update. The authors also report 15 training seeds and additional results in the appendices. These experiments provide useful evidence for the main claims.
  3. Analysis of reward flatness (Section 5, Tables 3 and 4, pp. 7 and 8, Appendix E.2, Tables 12 to 14, pp. 30.) The authors compare PPO and Mean SAM using a common parameter RMS dose. The measured flatness advantage becomes smaller across the four tasks, although Ant still shows a positive difference. The authors then compare return retention at similar action displacements. The results show no consistent advantage across tasks under this matching method.
Critical weaknesses
  1. Limited connection between theory and training (Sections 4.1 and 4.2, pp. 5 and 6, Appendix A.1, pp. 14 to 16, Appendix B.2, pp. 19 and 20) Proposition 1 analyzes one part of the feedback from the mean gradient to the policy scale. This part has a nonnegative total effect on the entropy direction. However, the total feedback contains other terms. The routing experiments test the full feedback from the mean to the scale. They do not measure the contribution of each term during native SAM training. The routing experiments also use gradient norm controls that differ from the native SAM update. The authors explain these limitations in the appendix. However, it remains unclear how much the result in Proposition 1 explains the scale changes observed during training. The theory supports the proposed explanation, but it does not fully explain the observed robustness gains.
  2. Limited analysis of scale history (Section 3.2, pp 4, Section 3.3, pp 5, Appendix B.2, pp 19, Appendix C.1, pp 23 to 25) The replay experiments use a different training procedure from native SAM. The scale follows a recorded history and remains fixed during each update cycle. Mean SAM also perturbs only the mean parameters. The authors report differences between self replay and native training in Appendix C.1. Therefore, the replay results support the role of scale history within this protocol, but they do not fully explain the robustness gains of native SAM.
  3. Limited analysis of the SAM perturbation radius (Section 3, pp 3, Section 4.1, pp 5, Appendix B.1, Table 5, pp 18) The main experiments use a SAM perturbation radius of 0.008. The paper does not report whether the main results remain similar at other radius values. The radius affects the SAM update and the feedback analyzed in Proposition 1. The results may depend on the selected radius. An additional experiment with another radius would help check whether the scale history remains more important than the mean updater.
  4. Fixed scale comparison uses only one scale value (Section 3.3, Figure 3, pp 5, Section 5.2, Table 3, pp 7, Appendix C.2, Table 8, pp 26) Main reward flatness experiments compare PPO and Mean SAM with the scale fixed at 0.20. However, Figure 3 shows that different training scales produce different returns and robustness results. The paper does not show whether the same reward flatness results hold at other fixed scale values. Testing another scale value would help check whether the results depend on the selected training scale.
  5. Limitations of action displacement matching (Section 5.3, Table 4, page 8, Appendix B.3, Appendix E.2, Tables 13 ) The matching method compares the average action displacement on shared states. It does not compare the direction of action changes or the resulting trajectories. The matched points also do not have the same parameter perturbation dose. The authors mention these limitations and provide additional tests. The results show no consistent advantage in return retention under this matching method. However, they do not establish that the two policies behave the same under other perturbations.
Questions for authors

o How was the SAM perturbation radius of 0.008 selected? Do the scale history replay results remain similar at another radius value? o Proposition 1 describes one part of the scale feedback. How large is this part compared with the other feedback terms during native SAM training? o The fixed scale experiments show that scale level affects robustness. Do the timing and scale of each action dimension also affect robustness? Can the authors explain this based on the reported results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vivb (1/3)

Thank you for the detailed review. Below we address each concern with the corresponding evidence. All intervals are 95%.

Short answers

  • Q1, SAM radius. We used ρ = 0.008 across all tasks, taking the HalfCheetah/Walker2d setting reported by Lee & Yoon as a reference and keeping one shared radius rather than tuning it per task. We additionally report a five-radius sensitivity sweep (0.002–0.032, n = 15) and repeat the replay experiment at ρ = 0.016 (point 3).
  • Q2, Prop 1's contribution. The ratio term cancels 85–94% of the cross term. Removing their sum reduces the final scale difference by 48–55% along the reported intervention path (point 1).
  • Q3, timing and dimensions. For matched 1M-step SAM windows, late placement gives higher final σ and nAUC point estimates. Per-dimension changes are largely a common factor; the independent effects of scale timing and dimensional structure are not isolated (point 6).
  • W2, replay vs. native training. Agreed. New scale-update interventions with σ learned online complement replay (point 2).
  • W4, another fixed scale. At σ = 0.40, A80 point estimates remain higher in all four environments, without a consistent action-AUC gain (point 4).
  • W5, displacement matching. Agreed: matching is limited to mean displacement magnitude (point 5).

1. Prop 1 and scale changes in training (W1, Q2)

Size. Fixed-checkpoint probes along SAM-PPO runs (15 seeds × 10 checkpoints × 32 minibatches per environment, before clipping) give coordinate-sum means, ×10⁻³: cross term +51.7 to +100.0, ratio term −47.7 to −93.8, and scale term −0.45 to −0.76. The full first-order feedback is +3.5 to +10.8, or 6–14% of the cross term. Its median cosine with the measured PPO − SAM gradient difference on φ is 0.995–0.999.

Training (general response, n = 10). Removing only the cross term from SAM-PPO puts mean final σ below PPO in all four environments; adding it as extra feedback to PPO gives 1.19–1.50× SAM-PPO's scale. Removing the mean-to-scale feedback (cross term + ratio term) removes 48–55% of the difference in mean final σGM\sigma_{\mathrm{GM}} between SAM-PPO and PPO. A further 13–34% disappears after removing clipping cross-talk; 18–32% remains unattributed. These are sequential terminal contrasts that depend on removal order. The general response expands Prop 1 and corrects its net-effect interpretation.

Routing controls. We compared ψ→φ with ψ→ψ under three protocols (n = 15). Under standard routing (full clean-gradient normalization, shared clipping), final σGM\sigma_{\mathrm{GM}} differences are +0.024 / +0.068 / +0.055 / +0.075 (HalfCheetah / Hopper / Walker2d / Ant), all with intervals above zero. All 12 scale-difference intervals across the three protocols are above zero. The paper's clipping-neutral protocol has contrasts 1.5–2.6× larger than the other protocols. Protocols span different code versions and 10 vs. 50 evaluation episodes; comparisons are paired within, not across, protocols (Table F, K9WT's thread). To first order, routing transmits both cross and ratio terms, not the cross term alone.

2. Replay differs from native training (W2)

Agreed. Replay prescribes σ from a recorded history, holds it fixed within each update, and applies Mean-SAM only to the mean. It identifies history effects within this controlled 2×2 protocol, not a fraction of native SAM-PPO's robustness gain.

Appendix C.1 reports self-replay/native differences. At ρ = 0.016, Mean-SAM self-replay minus native SAM-PPO has action-AUC differences −273 / −71 / +68 / +26, with all intervals containing zero; this is not an equivalence test.

New scale-update interventions with σ learned online (n = 10):

  • SAM mean updater, feedback removed. With clipping cross-talk removed in both arms, keeping rather than removing the mean-to-scale feedback raises final σGM\sigma_{\mathrm{GM}} by +0.052 to +0.114 and action AUC by +407 / +167 / +458 / +176. Both outcomes have intervals above zero in all four environments; nominal-return intervals contain zero. These arms retain SAM's mean-update rule and are paired by seed on one machine (general response 2/2, first and last columns).
  • PPO mean updater, feedback added. "PPO + cross term" has an action AUC 1.93–2.06× PPO's. Point estimates exceed SAM-PPO's in all four environments; difference intervals are above zero in Hopper and Ant. Comparisons with PPO/SAM-PPO references are unpaired across machines.
  • A counterexample to a simple scale–AUC mapping. After removing both mean-to-scale feedback and clipping cross-talk, final σ remains 1.5–2.1× PPO's. Yet action-AUC difference intervals contain zero in three environments; Ant is positive alongside a higher nominal return. This arm and PPO differ in both mean updater and scale.

These results connect the controlled replay finding to scale-update interventions with σ learned online.

AuthorsOpenReview ↗

Response to Reviewer vivb (2/3)

3. SAM radius (W3, Q1)

Choice. We used ρ = 0.008 across all tasks, taking the HalfCheetah/Walker2d setting reported by Lee & Yoon as a reference and keeping one shared radius rather than tuning it per task.

Five-radius sweep (ρ ∈ {0.002, 0.004, 0.008, 0.016, 0.032}, n = 15; not reported in the submission). Mean final σGM\sigma_{\mathrm{GM}} increases monotonically with ρ in all four environments. Action-AUC point estimates increase from 0.004 to 0.016 in all four; Hopper changes little between 0.002 and 0.004. At 0.032, action AUC declines in HalfCheetah and Ant alongside lower nominal returns. At 0.008, mean nominal return is highest among the five settings in HalfCheetah and Walker2d and near the highest in Ant, but not in Hopper.

Replay at a second radius (new; ρ = 0.016, seeds 1–10). The protocol is unchanged: the SAM scale history comes from SAM-PPO at ρ = 0.016, and Mean-SAM also uses 0.016.

Table G. Scale-history effect on action AUC: SAM history − PPO history under the same mean updater. Means [95% paired seed-bootstrap intervals] at ρ = 0.016; parentheses give the corresponding means at ρ = 0.008 on the same seeds.

EnvPPO mean updaterMean-SAM mean updater
HalfCheetah+890 [+542, +1241] (+695)+854 [+466, +1198] (+821)
Hopper+397 [+319, +475] (+291)+372 [+267, +467] (+346)
Walker2d+1392 [+1177, +1631] (+961)+1239 [+1076, +1413] (+901)
Ant+494 [+413, +577] (+408)+663 [+613, +729] (+496)
  • All eight scale-history intervals are above zero. All eight point estimates are larger than their counterparts at ρ = 0.008.
  • The mean-updater effect (Mean-SAM − PPO under the same history) ranges from −330 to +63 at ρ = 0.016. Six of eight point estimates are negative; three intervals are below zero.
  • At both tested radii, every scale-history effect is larger in point estimate than the absolute mean-updater effects in the same environment.

Thus, the larger effect of changing the scale history than changing the mean updater persists at a second radius within this replay protocol.

Radius and Prop 1. At a fixed parameter point, all three first-order terms are proportional to ρ. We measured their cancellation only at ρ = 0.008.

4. Another fixed scale (W4)

We compared Mean-SAM with PPO at fixed σ = 0.40 (seeds 1–10; flatness protocol of §5; not pre-registered). Mean-SAM perturbs only the mean parameters, with ρ = 0.008.

Table H. Mean-SAM − PPO at σ = 0.40, means [95% paired seed-bootstrap intervals]. The submitted σ = 0.20 comparison uses n = 15 (Table 3).

EnvΔA80Δ action AUCΔnAUCΔA80 at common parameter perturbation size
HalfCheetah+0.094 [−0.011, +0.188]−228 [−499, +24]−0.029 [−0.060, +0.002]+0.014 [−0.092, +0.103]
Hopper+0.016 [−0.147, +0.177]−166 [−365, −2]+0.278 [−0.010, +0.661]−0.070 [−0.236, +0.100]
Walker2d+0.216 [+0.047, +0.382]−146 [−379, +77]−0.058 [−0.161, +0.013]+0.143 [−0.030, +0.316]
Ant+0.186 [+0.124, +0.250]+72 [+34, +111]+0.003 [−0.012, +0.020]+0.039 [−0.027, +0.106]
  • Flatness: A80 point estimates are higher for Mean-SAM in all four environments, as at σ = 0.20. The intervals are above zero in Walker2d and Ant.
  • Action-noise robustness: there is no consistent gain across environments at either scale. At σ = 0.40, Ant has a positive action-AUC difference, alongside higher nominal return (+162 [+33, +293]); its nAUC difference interval contains zero.
  • Common parameter perturbation size: all four A80 difference intervals contain zero.

Three details matter for interpretation:

  • Under the looser displacement filter retaining 75% of PPO's displacement distribution, Walker2d and Ant retain positive A80 differences with intervals above zero. Under the stricter 25% and 50% filters, all four intervals contain zero.
  • In Hopper, 3/10 Mean-SAM seeds finish at low returns after late-training oscillations. All seeds are retained. Across all ten seeds, nominal return is lower than PPO's by −914 [−1505, −370], and action AUC is also lower. Low zero-noise returns in unstable seeds inflate their nAUC ratios.
  • A80 uses 80% of each model's own grid-center return as its threshold. It measures relative return retention, not absolute perturbed performance.
AuthorsOpenReview ↗

Response to Reviewer vivb (3/3)

5. Limits of displacement matching (W5)

Agreed. Matching controls only the average action-displacement magnitude on shared states. It does not match action-change directions, state-wise patterns, or subsequent trajectories; matched points also need not have the same parameter perturbation size.

For the submitted fixed-σ = 0.20 comparison, the six post hoc matching rules in Appendix E.2 give the same qualitative conclusion: no consistent matched return-retention advantage for Mean-SAM across tasks. This conclusion concerns the reported scalar matching, not broader behavioral equivalence.

The separate action-noise evaluation runs whole episodes in closed loop and includes the resulting state distributions and trajectories. Fixed-scale Mean-SAM shows no consistent gain across environments there either (point 4). External random action noise differs from parameter-induced action changes, so this evaluation complements displacement matching rather than replacing it.

6. Timing and per-dimension scale (Q3)

We provide two analyses not reported in the submission, both with n = 15.

Timing. During 3M environment steps, SAM is enabled only in the first or last 0.5M or 1M steps; PPO is used outside that window. All 16 window/environment combinations have lower action-AUC point estimates than full-training SAM-PPO, with difference intervals below zero in 15.

Table I. Late − early placement of the same 1M-step SAM window: last 1M minus first 1M steps. Means [95% paired seed-bootstrap intervals].

EnvΔ final σGM\sigma_{\mathrm{GM}}Δ nominal returnΔ action AUCΔnAUC
HalfCheetah+0.053 [+0.046, +0.059]−682 [−1463, +108]+33 [−168, +262]+0.063 [−0.006, +0.134]
Hopper+0.077 [+0.071, +0.084]+45 [−428, +521]+141 [+66, +213]+0.079 [−0.009, +0.162]
Walker2d+0.041 [+0.030, +0.052]−363 [−764, +102]+73 [−68, +211]+0.063 [+0.025, +0.100]
Ant+0.069 [+0.065, +0.073]−836 [−1293, −367]+6 [−74, +81]+0.066 [+0.046, +0.087]
  • Later placement raises final scale in all four environments, with intervals above zero. nAUC point estimates also rise in all four; its intervals are above zero in Walker2d and Ant. The action-AUC interval is above zero only in Hopper.
  • Early-only SAM ends at a lower scale than full-training SAM-PPO. For example, HalfCheetah ends at 0.060 versus 0.156 for full-training SAM-PPO and 0.041 for PPO.
  • For the 0.5M windows, all four late-minus-early nAUC intervals contain zero; Ant's action-AUC difference is negative (−118 [−168, −62]).

These results show that the timing of SAM-enabled training matters. Later 1M-step windows produce higher final scales alongside higher nAUC point estimates. Because the experiment switches the whole SAM update, it changes both mean and scale trajectories and does not isolate the effect of scale timing from final scale or mean-update effects.

Per-dimension scale (final checkpoints of the main batch; 23 action coordinates across four environments).

  • SAM-PPO's σ is higher than PPO's in all 15 seeds for every coordinate. Coordinate-wise geometric-mean ratios across seeds range from 3.2 to 4.9.
  • In log-scale, the component shared by all coordinates accounts for 92.5–97.4% of the squared change, averaged over seeds within each environment.
  • Between-coordinate log-scale spread has lower point estimates in all four environments; difference intervals are below zero in Walker2d and Ant and contain zero in HalfCheetah and Hopper.

Thus, most of the observed scale increase is shared across coordinates. Replay replaces the full per-dimension history, while fixed-scale experiments assign the same σ to every coordinate. Neither comparison isolates the effect of the dimensional profile while holding the overall scale history fixed.

Planned revision. We will clarify radius selection and replay's scope in §3, routing's scope in §4, and the matching limits in §5.3 and Appendix E.2. The appendix will add the radius sweep, Tables G–I, the per-dimension analysis, and the learned-scale training comparisons in point 2 during the discussion period.

Reviewer_K9WTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the role of policy scale in the robustness of SAM-PPO. The authors analyze how SAM affects the policy mean and scale during training. They also study the relationship between reward flatness and robustness, and propose RFD-PPO as an alternative method for improving robustness.

Critical strengths

The scale-history replay and fixed-scale experiments provide useful evidence that policy scale plays an important role in action robustness.

Critical weaknesses
  1. The experiments do not show that policy scale improves robustness independently of reward flatness. The flatness of scale-controlled PPO is not measured, and the Mean-SAM experiment does not directly test SAM-PPO. The reported flatness difference also largely disappears when the parameter perturbation size is matched.
  2. Proposition 1 only analyzes one component of the scale update. The positive cross term does not determine the direction of the full scale gradient. The gradient-routing experiments also change other parts of the update, so they do not directly verify the proposed mechanism.
Questions for authors
  1. How do reward flatness (A80 or λ+) and action robustness change across the fixed-scale PPO experiments in Table 8? Does increasing the training scale also produce flatter reward landscapes?
  2. How large are F_cross, F_ratio, and F_phi during training? Does using only F_cross in the scale update reproduce the scale increase observed in SAM-PPO?
4 comments
AuthorsOpenReview ↗

Response to Reviewer K9WT (1/2)

Thank you for the careful reading and the two precise questions.

Short answers

  • Q1. From σ = 0.05 to 0.20, A80 and action-AUC point estimates rise together in all four environments (Table C). We therefore do not claim that policy scale improves robustness independently of reward flatness. Table D examines the accompanying action displacements.
  • Q2. The coordinate-sum means are: cross term +51.7 to +100.0, ratio term −47.7 to −93.8, and scale term −0.45 to −0.76 (×10⁻³; Table A in the general response). The ratio term cancels 85–94% of the cross term. Adding only the cross term as additional feedback on top of PPO's own scale gradient gives a mean final scale 1.19–1.50× SAM-PPO's. Removing only the cross term from SAM-PPO puts the scale below PPO in all four environments (n = 10; Table B in the general response; Table E in post 2/2)
  • W1, Mean-SAM is not SAM-PPO. Agreed. It is a controlled mean-updater comparison. We also test a scale-feedback intervention with σ learned (point 2).
  • W1, matched perturbation size. Agreed. The reported flatness advantage depends on perturbation size; Ant retains a positive difference at σ = 0.20 (point 3).
  • W2. Agreed: the cross term's sign does not determine the full feedback, and routing does not isolate it. The general response gives the expanded analysis and training interventions; point 4 addresses routing.

1. Flatness across the fixed-scale sweep (Q1)

The submission did not report A80 for the sweep. Table C uses the same final checkpoints as Table 8 (n = 15).

Table C. A80 / action AUC of fixed-scale PPO. A80 is the fraction of grid points retaining at least 80% of the grid-center return. SE of A80: 0.03–0.07.

Envσ = 0.050.100.200.400.80
HalfCheetah.27 / 605.32 / 1301.46 / 2181.48 / 3297.80 / 1392
Hopper.26 / 488.29 / 742.45 / 920.68 / 1262.67 / 1362
Walker2d.14 / 562.22 / 881.35 / 1613.61 / 2333.84 / 2008
Ant.48 / 594.52 / 866.72 / 1252.71 / 1264.52 / 18
  • From σ = 0.05 to 0.20, both point estimates rise in all four environments. At 0.40 both rise further in the first three; in Ant, A80 is slightly lower and action AUC slightly higher.
  • At σ = 0.80, A80 must be read alongside performance: the grid-center return falls to 2938 in HalfCheetah and 48 in Ant. Its 80% threshold is relative to each model's own return.
  • λ+ was not measured on this sweep, so we answer with A80.

A parameter perturbation first changes the policy's mean actions, and return is then evaluated under the perturbed policy. To interpret A80, we examine the resulting action displacement and the fraction of grid points preserving return within a shared displacement band.

Table D. Mean full-grid action displacement / A80 within a shared displacement band. The band is the 25–50% quantile range of displacement at σ = 0.20; within-band mean displacement differs by at most 0.03 across scale levels. All levels use the same relative-grid protocol. n = 15; SE of banded A80: 0.02–0.08.

Envσ = 0.050.100.200.40
HalfCheetah.224 / .04.292 / .18.356 / .53.420 / .70
Hopper.136 / .24.127 / .23.162 / .54.208 / .83
Walker2d.180 / .03.268 / .14.318 / .36.379 / .70
Ant.195 / .13.263 / .37.344 / .87.425 / .96
  • Full-grid mean displacement is higher at σ = 0.40 than at 0.05 in all four environments; Hopper has a small dip at 0.10.
  • Within the shared band, A80 rises from σ = 0.10 to 0.40 in all four environments. The other three bands show the same trend over this range; at σ = 0.05, the highest band in Walker2d is an exception.
  • σ = 0.80 is omitted from this comparison because Ant has collapsed.

Thus, the higher A80 across these scale levels is accompanied by larger, not smaller, full-grid action displacements. Within the shared displacement band, higher σ is associated with more grid points preserving return. The band restricts displacement magnitudes, not directions or trajectories. It is consistent with the action-noise results, while evaluating parameter-induced shifts rather than external Gaussian noise.

Post 2/2 addresses Mean-SAM's scope, perturbation-size matching, and routing.

AuthorsOpenReview ↗

Response to Reviewer K9WT (2/2)

2. Mean-SAM does not directly test SAM-PPO (W1)

Agreed. Mean-SAM freezes φ and applies SAM only to ψ. It tests whether changing the mean updater at fixed σ improves action-noise robustness. Native PPO/SAM-PPO comparisons cannot isolate this: SAM-PPO's scale is about four times PPO's. Intervals below are 95%.

  • σ = 0.20 (Table 3, n = 15): A80 point estimates are higher in all four environments. The action-AUC difference (Mean-SAM − PPO) is −200 [−345, −68] in HalfCheetah; intervals contain zero in the other three.
  • σ = 0.40 (new, n = 10, not pre-registered): A80 point estimates remain higher in all four. The action-AUC interval is above zero only in Ant (+72 [+34, +111]), alongside higher nominal return (+162) and a small nAUC difference (+0.003). Hopper has late-training oscillations in 3/10 Mean-SAM seeds; the all-seed differences are −914 [−1505, −370] in nominal return and −166 [−365, −2] in action AUC.

With σ learned, "PPO + cross term" keeps PPO's mean-update rule, applies no parameter perturbation, and adds the cross term as extra scale feedback.

Table E. PPO / SAM-PPO / PPO + cross term, n = 10. PPO and SAM-PPO are from the main batch on another machine; comparisons are unpaired.

EnvFinal σGM\sigma_{\mathrm{GM}}Action AUC
HalfCheetah.042 / .155 / .1971066 / 1812 / 2075
Hopper.054 / .220 / .317491 / 791 / 948
Walker2d.068 / .282 / .336865 / 1772 / 1782
Ant.039 / .168 / .251694 / 1205 / 1344
  • Action AUC is 1.93–2.06× PPO's. Point estimates exceed SAM-PPO's in all four environments; difference intervals are above zero in Hopper (+157 [+33, +281]) and Ant (+138 [+87, +186]).
  • Nominal-return differences from SAM-PPO have intervals containing zero in three environments; Ant is lower (−287 [−463, −102]).
  • Against PPO in Hopper and Walker2d, nominal-return intervals contain zero, while nAUC is higher by 0.33 and 0.34 (intervals above zero). The normalized gains indicate improved relative performance under noise.

Thus, modifying the scale update improves action-noise robustness without SAM parameter perturbations. SAM's mean update may still contribute in native SAM-PPO.

3. The flatness gap after matching perturbation size (W1)

Agreed; the paper reports this. On common-size grids, perturbation size is matched to same-seed PPO. ΔA80 (Mean-SAM − PPO, σ = 0.20, n = 15) is −0.017, −0.052, and −0.050, with intervals containing zero, in HalfCheetah, Hopper, and Walker2d; Ant retains +0.133 [+0.052, +0.218]. At σ = 0.40 (n = 10), all four intervals contain zero.

On relative grids at σ = 0.20, Mean-SAM's higher A80 accompanies smaller mean action displacement (by 0.071 in HalfCheetah and 0.121 in Ant; intervals contain zero in Hopper and Walker2d, as do their unmatched ΔA80) and smaller weights (mean-network spectral-norm product 31–50% smaller, seeds 1–10). Size matching changes ΔA80 from +0.091 to −0.017 in HalfCheetah and +0.251 to +0.133 in Ant. Ant's action-AUC difference is +22 [−25, +74].

Together with point 1, the two settings show different patterns. Across moderate fixed scales, higher A80 generally accompanies larger displacements and higher within-band A80 and action AUC. Mean-SAM's A80 advantage accompanies smaller displacements and weights, shrinks under size matching, and has no consistent action-AUC gain. Endpoint A80 alone therefore cannot identify the training pathway behind robustness.

4. Routing also changes other parts of the update (W2)

Agreed. To first order, route ψ→φ transmits the mean-to-scale feedback (cross term plus ratio term, without the scale term), not the cross term alone. The paper's protocol also gives the full SAM radius ρ to ψ, rescales the receiving gradient to its unperturbed clipped norm, and neutralizes clipping. We compared ψ→φ and ψ→ψ under three protocols (n = 15, paired by seed within each protocol).

Table F. ψ→φ − ψ→ψ: Δ final σGM\sigma_{\mathrm{GM}} / Δ action AUC. † = 95% interval above zero.

EnvStandard (no control)+ full radius ρ to ψ+ clipping neutral (paper)
HalfCheetah+0.024 / +215+0.021 / +104+0.055 / +321†
Hopper+0.068 / +93+0.067 / +94†+0.125 / +252†
Walker2d+0.055 / +153+0.053 / +155+0.123 / +282†
Ant+0.075 / +229†+0.073 / +149†+0.114 / +287†

All 12 Δσ intervals are above zero, including under the standard protocol. The contrast is larger under the paper's clipping-neutral protocol. The first two protocols come from earlier batches with different code versions and 10 evaluation episodes per noise level (vs. 50 in the paper); columns are not paired across protocols.

Planned changes. Add Tables C–F to the appendix, revise §5 around points 1 and 3, and state in §4 what routing transmits to first order.

Thanks for running the additional experiments. Table C answers my question about the fixed-scale PPO runs. The gradient breakdown and cross-term ablations address my concern about Proposition 1. In the paired experiment, both groups used the same SAM mean updater and had clipping cross-talk disabled. Keeping the scale feedback improved action AUC in all four tasks.

I'm still unsure how much of the robustness gain in native SAM-PPO comes from this feedback. At moderate fixed scales, both A80A_{80} and action AUC increase in Table C. That leaves open whether reward flatness contributes to the gain. I'd like the revision to make that clear. I'll keep my rating at 2.

AuthorsOpenReview ↗

Follow-up to Reviewer K9WT

Thank you for the follow-up. We have uploaded the revised manuscript and made both points explicit.

Reward flatness. We agree that the rise in both A80A_{80} and action AUC at moderate fixed scales leaves open a role for reward-surface flatness within the scale pathway (§1 and §7). The results from Table C are discussed in §5.2 and reported in Appendix C.2, Tables 13–14. Two fixed-scale controls clarify what a higher relative-grid A80A_{80} tells us:

  • Mean-SAM. At σ=0.20\sigma=0.20, matching parameter perturbation size to same-seed PPO reduces the mean A80A_{80} difference across the four tasks from +0.131+0.131 to +0.004+0.004. A positive difference remains in Ant (§5.2; see also the σ=0.40\sigma=0.40 comparison in Appendix E.4).
  • Weight-norm control without SAM. At fixed scale, L2L_2 regularization yields smaller parameter perturbations and higher relative-grid A80A_{80} in all six comparisons, without consistent action-AUC gains. After matching perturbation size, five of the six A80A_{80} difference estimates are negative (§5.4 and Appendix E.3).

Thus, higher relative-grid A80A_{80} alone does not imply greater action-noise robustness.

Share of native SAM-PPO’s gain. In paired runs with the same SAM mean-update rule and clean-gradient clipping in both arms, retaining mean-to-scale feedback increases action AUC by +407/+167/+458/+176+407 / +167 / +458 / +176 in HalfCheetah, Hopper, Walker2d, and Ant, respectively. All four 95% confidence intervals are above zero (Appendix A.4, Table 9).

For comparison, after cancelling the feedback and removing clipping cross-talk, final σGM\sigma_{\mathrm{GM}} remains 1.5–2.1 times PPO’s. The corresponding action-AUC differences from PPO are +39/+91/+172/+257+39 / +91 / +172 / +257. These comparisons are unpaired across machines: the 95% confidence intervals include zero in the first three tasks, while Ant’s interval is above zero alongside higher nominal return (Appendix A.4, p. 23).

The paired result measures the effect under clean-gradient clipping. Changing the feedback also changes subsequent Adam updates and rollouts, so this effect does not give an additive share of the native action-AUC gain. The separate 48–55% figure measures an order-dependent reduction in the final SAM-PPO–PPO scale gap under native clipping, not a fraction of the action-AUC gain (Appendix A.4).

Reviewer_y6cHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the previously reported benefits of applying Sharpness-Aware Minimization (SAM) to PPO. The SAM-PPO procedure perturbs the actor parameters in a direction that increases the loss, then uses the perturbed parameters to compute the update gradient, arguably making the policy robust to small action errors. This paper argues that those benefits instead come from SAM-PPO maintaining a higher standard deviation of actions during training.

Critical strengths
  • I like the motivation behind this paper: picking apart a prior explanation of an observed phenomenon and investigating where these benefits actually come from.
  • I also find the experiment of using the learned standard deviation schedule to PPO innovative
Critical weaknesses
  • I find the paper difficult to follow because of unnecessary jargon. The key ideas aren't that complicated, but the text structure, verbosity, and flow make it hard to understand what they are trying to do. I generally struggled to connect each hypothesis to its corresponding figure/results, and found quite some terms either undefined or not concretely connected to hte narrative; some examples of which are:
    • Policy scale/training scale / learnable scale are not consistently connected to the narrative.
    • Parameter dose – what does that mean?
    • common-scale reading
  • The write-up relies heavily on SAM-PPO intuition: although it mechanically describes the SAM update, it explains little about why one should expect reduced sensitivity to parameter perturbations, which should improve tolerance to action noise. This matters because the paper centrally challenges this idea.
  • I am not sure RFD has a practical advantage over simply fixing the standard deviation during PPO training. When I compare PPO at σ=0.40\sigma=0.40 in Table 8 to RFD in Table 2, PPO with this setting outperforms RFD in 3/4 envs tested.
  • Proposition 1 identifies a component of SAM’s additional update that encourages greater overall policy randomness. However, the complete update also includes effects from changes in PPO’s probability ratios and from perturbing the log-standard-deviation parameters. Although Appendix A.1 acknowledges this gap, I think it is important to empirically establish what the proposition argues, and the current routing experiments do not isolate it. I would like to see the predicted contribution compared with the total difference between the SAM and PPO update directions for the log standard deviations, using the same actor checkpoint and minibatch. My recommendation to hte authors: compute the ordinary PPO gradient for hte log-std-dev, then compute hte SAM gradient for those same parameters, and then calculate the particular contribution proposed in Proposition 1. Comparing its magnitude na direction with hte total difference between SAM and PPO should help empirically establish this. An additional ablation removing only this component would help establish whether it explains SAM’s higher training standard deviation

Overall, although I like the motivation and hte idea, I think the paper needs a larger rewrite, which is not possible during the rebuttal.

Questions for authors

Please see the weaknesses section

2 comments
AuthorsOpenReview ↗

Response to Reviewer y6cH

Thank you for the careful reading and the test protocol for Prop 1. It exposed an error in our interpretation.

Short answers

  • W1, terms. Agreed. We define the three terms below and will use one name per quantity.
  • W2, the argument we examine. We study how SAM-PPO acquires action-noise robustness, separating policy-scale and mean-update effects.
  • W3, RFD vs. fixed σ. Your comparison is correct. RFD is a mechanism-testing control, not a proposed method.
  • W4, Prop 1. We ran your protocol and ablation. The ratio term cancels 85–94% of the cross term; the cross term's nonnegative sum does not determine the full feedback.
  • W5, rewrite. We agree that the presentation needs substantial revision.

W1. Terms and structure

In the submissionMeaningIn the revision
policy scale / training scale / learnable scaleThe Gaussian policy's state-independent standard deviation, σj=exp⁡(ϕj)\sigma_j=\exp(\phi_j), one per action dimension"policy scale"; learned / fixed / replayed specify how σ is set
parameter doseRMS Euclidean size of parameter perturbations on an evaluation grid (DrmsD_{\mathrm{rms}}, Appendix A.2, Eq. 3)"perturbation size", defined at first use
common-scale readingLargest eigenvalue λ+ of the PPO-loss Hessian with respect to mean parameters, evaluated at a shared σ (Appendix C.3)"λ+ at a shared σ"

Actions are sampled using σ during training. Action-noise evaluation uses the mean plus external noise, without sampling from σ. Evaluating λ+ at a shared σ involves no retraining and differs from the reward-flatness measure A80. Each section will open with its hypothesis and the figures and tables that test it.

W2. SAM-PPO, parameter perturbations, and action noise

We study how SAM-PPO acquires action-noise robustness. The prior flatness-based interpretation is:

  1. SAM minimizes the worst-case PPO surrogate loss in a parameter neighborhood, favoring lower sensitivity to parameter perturbations.
  2. With covariance fixed, a small mean-parameter perturbation induces a shift in the policy mean through its Jacobian (Lee & Yoon, 2025, Appendix A, Eqs. 10–13).
  3. Return preserved under parameter perturbations is therefore preserved under the action shifts they induce.

This connects parameter perturbations to action shifts, but those structured, state-dependent shifts need not coincide with the external Gaussian action noise used in evaluation. This local mapping does not identify the training mechanism.

In Gaussian SAM-PPO, SAM updates both the mean and scale. Scale changes the training action distribution. We separate scale-history and mean-updater effects:

  • Scale interventions. Replaying SAM-PPO's scale history improves PPO's action AUC in all four environments without SAM updates (Fig. 2). In fixed-scale PPO, A80 and action AUC both rise from σ = 0.05 to 0.20 in all four environments (Table C, K9WT's thread).
  • Mean-update control. At fixed σ = 0.20, Mean-SAM has higher A80 point estimates in all four environments but no consistent action-AUC gain (Table 3).

These interventions identify a scale pathway behind SAM-PPO's robustness. Endpoint flatness alone cannot identify how that robustness was acquired. Geometry may still participate in this pathway.

W3. RFD vs. fixed σ

Action AUCHalfCheetahHopperWalker2dAnt
PPO at fixed σ = 0.40 (Table 8)3297126223331264
RFD (Table 2)2205113320331286

Your comparison is correct: fixed-σ PPO is higher in three environments; Ant is the reverse. Like entropy regularization, RFD tests the scale pathway: σ remains learned, a restoring term acts on the scale, and the mean is updated without SAM. Robustness improves under this control. Here σ = 0.40 is selected from the sweep and the best level differs by task; RFD uses one rule across tasks. These results do not establish which is more practical. We will clarify RFD's role and include this comparison in §4.3.

W4. Prop 1 (Tables A and B in the general response)

  • Same checkpoint, same minibatch. The ratio term cancels 85–94% of the cross term. The median cosine with the measured pre-clipping PPO − SAM gradient difference on φ is −0.14 to +0.37 for the cross term and 0.995–0.999 for the full three-term sum.
  • Your ablation and its reverse. Removing only the cross term from SAM-PPO puts mean final σGM\sigma_{\mathrm{GM}} below PPO in all four environments (n = 10). Adding only the cross term to PPO gives a mean final scale 1.19–1.50× that of SAM-PPO.
  • Routing. Agreed: to first order, routing transmits the cross term together with the ratio term and does not isolate it.

W5. Revision

During the discussion period, we will expand Prop 1 into (i)–(iii), correct the interpretation of its cross term as the net effect, make the W1–W3 changes above, and add a glossary and the new results to the appendix.

AuthorsOpenReview ↗

Follow-up to Reviewer y6cH

Thank you again for suggesting the same-checkpoint, same-minibatch comparison. The revised manuscript reports it in §4.2 and Appendix A.3, with the cross-term ablation in Appendix A.4. The revised Proposition 1 (§4.1) retains the original cross-term result and makes clear why it does not determine the sign of the full feedback.

We also standardized the terminology and defined the terms where they are used. Section 1 now distinguishes the action shifts induced by parameter perturbations from the external action noise used at evaluation. Section 4.3 clarifies RFD’s role as an experimental scale control and includes the direct comparison with PPO at fixed σ=0.40\sigma=0.40.

The summary of manuscript revisions lists the other changes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the three reviewers for their careful reading. All three note that Prop 1 characterizes only the cross term, while the full feedback on the log-scale φ = log σ also includes a ratio term (from the dependence of the probability ratio on φ) and a scale term (from perturbing φ itself). We agree: the nonnegative coordinate sum in Prop 1 does not determine the sign of the full feedback. We will correct the interpretation in §1 and §4.1. Following y6cH's same-checkpoint, same-minibatch protocol and suggested ablation, and Q2 of K9WT and vivb, we find:

  • Full scale feedback. The ratio term cancels 85–94% of the cross term. The full feedback is aligned with the measured feedback (median cosine 0.995–0.999); the cross term alone has much weaker alignment (−0.14 to +0.37).
  • Training interventions. Removing only the cross term from SAM-PPO puts the final σ below PPO in all four environments. Adding only the cross term to PPO gives a mean final σ 1.19–1.50× that of SAM-PPO. Removing the mean-to-scale feedback (cross term + ratio term) from SAM-PPO removes 48–55% of the difference in final σ between SAM-PPO and PPO.
  • What does not depend on Prop 1. Scale-history replay tests the role of the scale history in action-noise robustness without using Prop 1. Two new results add to it (end of this post).

1. Same checkpoint, same minibatch

Along SAM-PPO training runs (15 seeds × 10 checkpoints × 32 minibatches per environment, m = 64), we computed the three first-order terms and compared their sum with the measured feedback: the difference between the unperturbed (PPO) and perturbed (SAM) gradients on φ, gϕ−hϕg_\phi - h_\phi, obtained by automatic differentiation at ρ = 0.008. All quantities are computed at the start of an update, where r = 1, and before gradient clipping.

Table A. Means of coordinate sums, ×10⁻³; positive values denote scale-increasing contributions. Cosines are between action-dimension vectors within each minibatch; medians are shown.

EnvCrossRatioScaleFullMeasuredcos(cross, measured)cos(full, measured)
HalfCheetah+100.0−93.8−0.45+5.7+6.6+0.130.995
Hopper+51.7−47.7−0.46+3.5+3.3+0.370.998
Walker2d+58.5−49.8−0.76+7.9+7.6+0.150.999
Ant+99.8−88.5−0.49+10.8+14.5−0.140.999

The scale term is 0.4–1.3% of the cross term. The full feedback has a positive mean in all four environments, with 95% seed-bootstrap intervals above zero.

2. Training interventions

Both arms change only the scale-update rule and run for 3M steps.

Table B. Mean final σGM\sigma_{\mathrm{GM}} (geometric mean of the policy scale over action dimensions), n = 10 seeds.

EnvPPOSAM-PPOSAM-PPO − cross termPPO + cross term
HalfCheetah0.0420.1550.0110.197
Hopper0.0540.2200.0090.317
Walker2d0.0680.2820.0300.336
Ant0.0390.1680.0130.251
  • SAM-PPO − cross term: in each environment, every seed ends below PPO's minimum. Mean nominal return is 22–36% of SAM-PPO's, so we do not use this arm's action AUC as evidence about robustness.
  • PPO + cross term: no SAM parameter perturbation is applied; the mean block follows PPO's update rule.
  • Removing the mean-to-scale feedback (Table B, continued in post 2/2): of the difference in final σ between SAM-PPO and PPO, 48–55% disappears after removing these two terms, a further 13–34% after also removing the clipping cross-talk (defined in post 2/2), and 18–32% remains unattributed. The split depends on the order of removal; it is not an additive decomposition.

Post 2/2 gives the revised Prop 1 and the full intervention table.

What does not depend on Prop 1, and what the new runs add

Our main claim, that action-noise robustness largely follows the scale history while the mean updater has smaller, task-dependent effects, rests on scale-history replay rather than Prop 1. The new runs provide two further tests:

  • Training without parameter perturbation. "PPO + cross term" has an action AUC 1.93–2.06× PPO's. Its AUC point estimates exceed SAM-PPO's in all four environments; the intervals of the difference are above zero in Hopper and Ant. Nominal return is lower than SAM-PPO's in Ant (−287 [−463, −102]); the 95% intervals contain zero in the other three environments.
  • A second SAM radius. Replay at ρ = 0.016 gives a positive scale-history effect in 8/8 comparisons, all with intervals above zero (vivb's thread).

PPO and SAM-PPO references are seeds 1–10 of the main batch on another machine, so comparisons with the intervention arms are unpaired.

We answer the other questions directly in each reviewer's thread and will upload a revised PDF during the discussion period.

AuthorsOpenReview ↗

General response (2/2)

Revised Prop 1

Write gg for the unperturbed actor gradient, G=∥g∥G=\|g\|, qt=A^t/mq_t=\hat A_t/m, and F=Fcross+Fratio+FϕF=F_{\mathrm{cross}}+F_{\mathrm{ratio}}+F_\phi for the first-order scale feedback; jj indexes action dimensions.

(i) Unchanged: on every minibatch, the coordinate sum of the cross term is nonnegative. All 19,200 minibatches behind Table A satisfy it.

(ii) At r = 1, fix states and advantages and assume that the standardized exploration noise is i.i.d. standard Gaussian. Then

=-\mathbb E[GF_{\mathrm{ratio},j}], \qquad \mathbb E[GF_{\phi,j}]=-4\rho\sum_tq_t^2.$$ So $\mathbb E[GF_j]=-4\rho\sum_tq_t^2\leq0$. Before division by G, the ratio term cancels the cross term in expectation, leaving the nonpositive scale term. **(iii)** SAM divides by the G of the same minibatch. For G > 0 and finite expectations, $$\mathbb E[F_j] =\mathbb E[GF_j] \mathbb E[1/G] +\operatorname{Cov}(GF_j,1/G).$$ Under (ii), the first term equals $-4\rho\sum_tq_t^2 \mathbb E[1/G]\leq0$. The expected feedback is positive if and only if the covariance contribution exceeds the magnitude of this nonpositive term. Division by each minibatch's own G reweights the feedback; its average sign is not fixed by (ii). **Scope.** All checkpoint probes hold parameters fixed at r = 1. During training, r = 1 holds exactly only before the first parameter update for each rollout. GAE advantages can depend on sampled actions, so the independence in (ii) is not guaranteed. The empirical decomposition therefore uses measured means in the exact identity (iii), rather than substituting the idealized mean from (ii). We checked (ii) by Monte Carlo on synthetic data satisfying its assumptions. **Real checkpoints** - Before division by G, the ratio term cancels 99–110% of the cross term; after division, 85–94% (Table A). - Decomposing across the 32 minibatches within each checkpoint gives the following coordinate-sum contributions (×10⁻³). The covariance contribution is positive in all four environments (+8.5 to +23.2; all 95% seed-bootstrap intervals above zero). The first term has negative point estimates in all four (−17.5 to −0.6), with intervals below zero in HalfCheetah and Hopper. The covariance contribution outweighs it in these probes. - In a separate scan at m = 256 and 1024, the full feedback is negative in Hopper (intervals below zero) and has negative point estimates in HalfCheetah (intervals contain zero). In Hopper, the covariance contribution stays positive but no longer offsets the first term. **Full intervention table** "Clipping cross-talk" refers to SAM-PPO computing the global clipping coefficient from the perturbed gradient h, which changes the coefficient applied to φ's gradient before Adam. "− clipping cross-talk" instead uses the coefficient from the unperturbed gradient g for φ. "− mean-to-scale feedback" removes the cross term plus the ratio term, per minibatch and action dimension. Coefficients are fixed before the intervention; modified gradients are not clipped again. *Table B, continued.* Mean final $\sigma_{\mathrm{GM}}$, n = 10. All arms use SAM-PPO with the listed changes to φ's update; the mean block retains the SAM update rule. Comparisons among these arms are paired by seed on one machine. | Env | − clipping cross-talk | − cross term − clipping cross-talk | − mean-to-scale feedback | − mean-to-scale feedback − clipping cross-talk | |---|---:|---:|---:|---:| | HalfCheetah | 0.126 | 0.045 | 0.093 | 0.074 | | Hopper | 0.173 | 0.012 | 0.140 | 0.083 | | Walker2d | 0.238 | 0.053 | 0.170 | 0.124 | | Ant | 0.139 | 0.037 | 0.097 | 0.080 | - Removing the mean-to-scale feedback lowers final σ under both clipping rules: relative to SAM-PPO (Table B; unpaired across machines; 48–55% of the final SAM–PPO scale gap), and relative to "− clipping cross-talk" (paired by seed). All eight 95% intervals are above zero. - After both removals (last column), final σ remains 1.5–2.1× PPO's. This part is unattributed: the scale term, finite-radius effects, and the SAM update of the mean remain in this arm. - **A prediction that failed.** We predicted that the second column would fall below "SAM-PPO − cross term" (Table B), but it is above it in all four environments, narrowly in Hopper (0.012 vs 0.009). The assumed ordering of clipping coefficients reverses late in the initial five-seed logs; the coefficient from g falls to 0.02–0.08 in HalfCheetah, Walker2d, and Ant. These coefficients rescale gradients before Adam, not the parameter steps directly. **Planned changes** We will expand Prop 1 in §4.1 into (i)–(iii), correct the §1 statement that mean perturbations "generate mean-to-scale feedback that opposes scale contraction" and the corresponding interpretation in §4.1, and add the proof, Tables A and B, and the minibatch-size sensitivity table to Appendix A.
AuthorsOpenReview ↗

Summary of manuscript revisions

We thank Reviewers y6cH, K9WT, and vivb for their reviews and constructive discussion. The revised manuscript is now available on the submission page, with key additions and revisions highlighted in blue.

The research question and main empirical findings are unchanged. We retained all original figures and tables. The plotted results and the numerical entries in Tables 1–4 are unchanged. The original scale-history replay and fixed-scale comparisons remain the basis for the paper’s main conclusion: policy scale during training shapes action-noise robustness, and higher endpoint flatness on relative grids does not by itself imply greater robustness. We clarified the interpretation of these results and added the follow-up analyses discussed with reviewers. The main text is 10 pages, and all new tables (Tables 5–9, 14, 16–19, 21, and 28) are in the appendix.

Scale feedback (§§4.1–4.2; Appendices A.1, A.3–A.4, and D.1). Proposition 1 retains the original cross-term result and now accounts for all three feedback terms. The nonnegative cross-term sum alone does not determine the sign of the full feedback. We added checkpoint probes that examine the role of minibatch normalization and paired training interventions that test the effect of retaining mean-to-scale feedback. With the same mean updater and clean-gradient clipping in both arms, retaining this feedback increased final policy scale and action AUC in all four tasks. Appendix D.1 also compares feedback routing across three norm-control protocols.

Scale-history replay (§3, §3.2; Appendices C.4–C.5). At a second SAM radius (ρ=0.016\rho=0.016), replaying SAM-PPO’s scale history instead of PPO’s again improved action AUC under either mean updater. The appendix also reports the radius sweep and the timing and per-dimension analyses.

Endpoint flatness (§§5.2–5.4 and §7; Appendices C.2 and E.4). The added geometry analysis and the Mean-SAM comparison at fixed σ=0.40\sigma=0.40 clarify when relative-grid flatness and action AUC move together and when they diverge. Reward-surface flatness may contribute within the scale pathway. We also clarify that action-displacement matching controls mean magnitudes, not directions or trajectories.

We standardized terminology and clarified RFD’s role as an experimental scale control (§4.3). For existing appendix results, we updated the routing confidence intervals (Appendix D.1) and corrected one rounded mean action displacement (Appendix F.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.